reconciliation
Reconciles the share database against the state of the storage.
- Package
github.com/cs3org/reva/v3/pkg/reconciliation- Source
- pkg/reconciliation · Go reference
- Used by
- reconciliation
Overview
Package reconciliation reconciles the share database against the state of the storage. It holds two jobs: orphan detection, which marks the shares and public links whose resource or recipient is gone, and the shallow check, which repairs the ACLs on the paths that carry a share. The engine is storage-driver agnostic: it reads shares from the database and resolves resources, identities and providers through the gateway (CS3).
Every line a job logs carries an “event” naming what happened, so a run can be replayed or reverted by filtering on it rather than by parsing free-form messages. An event is the job’s own name followed by the step: every job has start, skip, fail and end, plus one event for the change it makes. Each line also carries “job” and “run”, the uuid of the run it belongs to, so jobs stay apart even when they log to the same file.
The jobs span the shares and the public links, so they are wired as their own
serverless service rather than by one of the grpc share services. They go
through a share manager and a public share manager, named by driver like
everywhere else. Marking an item orphaned is not part of the CS3 API, so a
driver that does not implement it is refused at startup; today only sql does,
which is why it is the default for both.
Each job is enabled and configured on its own, so one can be left in dry-run, or rescheduled, or pointed at another log, without touching the other.
[serverless.services.reconciliation]
jobs = ["orphan", "shallow", "expired"]
share_driver = "sql"
publicshare_driver = "sql"
service_user_name = "cboxreco"
service_user_uid = 12345
service_user_gid = 2766
[serverless.services.reconciliation.orphan]
schedule = "@daily" # "@every <dur>" | "@hourly" | "@daily" | "@weekly"
dry_run = false
log_file = "/var/log/revad/reconciliation-orphan.log"
spaces_per_batch = 500 # omit to read everything in one go
[serverless.services.reconciliation.shallow]
schedule = "never" # or a cadence, on-demand runs work either way
dry_run = true
log_file = "/var/log/revad/reconciliation-shallow.log"
spaces_per_batch = 500 # omit to read everything in one go
[serverless.services.reconciliation.expired]
schedule = "@hourly"
dry_run = false
log_file = "/var/log/revad/reconciliation-expired.log"
A job runs only if it is listed in jobs. orphan and expired need a schedule; on
shallow it is optional and also takes never, since that job can be triggered
on demand whether or not it has a cadence. log_file defaults to the path shown
above for each, and both drivers default to sql, so all four keys above the
service user can be omitted.
spaces_per_batch limits one batch to a given number of spaces.
This has been introduces to avoid memory spikes due to large numbers of
shares being loaded into memory at the same time. If no value is set,
all shares will be loaded into memory at the same time.
With a batch size, a row whose space_id is NULL is not visited: there is no
space to put it in a batch under. The hierarchy check the share API runs already
skips such a share for the same reason.
The drivers take their configuration from
[serverless.services.reconciliation.share_drivers.<name>] and
[serverless.services.reconciliation.publicshare_drivers.<name>], the same keys
the usershareprovider and the publicshareprovider pass. The sql driver reads
its database settings from [shared], so neither section is normally needed,
and neither is jwt_secret.
Identity
The jobs runner hands a run a bare context, but we need a valid auth so we can do
stat’s etc. Therefore, each run mints itself a token for service_user_name and
sends it with every call. The three service_user_* keys are required and the
account has to be a real one: EOS reads the ACLs of a node as the caller before
handing them out, and its driver refuses a caller whose uid or gid is zero, so
root does not work (for CERNBox, cbox does the trick). If
skip_user_groups_in_token is set, the account also has to resolve in the user
provider, since the auth interceptor looks its groups up.
Jobs
reconciliation.orphans marks the shares and public links whose resource or
recipient no longer exists. With a spaces_per_batch it asks both stores which
spaces they hold an item in and reads those spaces a batch at a time. Each item
is judged on its own, so a batch is a listing and nothing more. If a batch cannot
be listed the run fails there: the job is idempotent, so what it marked stands
and the next run picks up the rest.
reconciliation.shallow writes the ACL entry each share implies onto its path,
when the storage has no entry or the wrong permissions. On the storage it only
adds and corrects entries. It never removes an ACL entry. In the share database
it does remove rows: a share that a higher share of the same recipient makes
redundant is deleted (soft). See below for the details.
reconciliation.expired removes the ACL entries of the shares whose expiration
date has passed. The share manager hides an expired share from the recipient,
but its ACL entry stays until this job runs. For each expired share it removes
the entry, reapplies the entries of the same recipient’s live shares above and
below it (as RemoveShare does), and sets expired_at on the row. The row is
not deleted, so the owner still sees the share. A share with expired_at set
is not picked up again. If a step fails, expired_at stays empty and the next
run tries again.
Start a run with reva admin jobs run reconciliation.shallow [space=<space-id>].
Leave the space out to cover every space. This needs the admin API enabled. The
command prints a run id: reva admin jobs status <run-id> shows the state and
the totals, reva admin jobs cancel <run-id> stops the run.
With a schedule, the job also runs on a cadence over every space. That run has
its own name, so reva admin jobs trigger reconciliation.shallow.scheduled
starts it before its next fire.
With a spaces_per_batch, a full run asks the database which spaces hold a share
and reads them a batch at a time. It decides and applies one space at a time
either way, so a run scoped with space=<space-id> is that same work over one
space.
A run works out the changes of a space before it writes any of them, and a user
can share in the meantime. So the shares of a space are listed again right before
its changes go out. A space that gained a share since the run listed it is left untouched,
whole, and picked up by the next run: the new share is not in what the run
compared, and it can be what makes a share the run wants to remove no longer
redundant. Only the spaces the run has something to change in are looked at
again. Their ids come back in the run’s totals under skipped_spaces, so a
retry can be scoped to exactly the spaces that were held back.
A share below another share of the same recipient is redundant when it grants the
same access (covered in the totals) or less (conflicting). The job deletes
the row of such a share, which is what the share API does when the higher share
is created. The delete is soft, so you can undo it in the database. The ACL entry
on the path stays.
A share whose expiration date has passed is left out (expired in the totals).
The job does not write its entry, and it does not make a share below it
redundant. The share API ignores expired shares in the same way.
Logs
Each job writes its own log (well, in fact, more like a journal), separate from revad’s. This is always written in JSON so that it is easy to parse by any tool in case we need to revert certain actions.
An event is the job’s own name followed by the step, so one job’s lines never
have to be told apart from the other’s by hand.
| event | meaning |
|---|---|
reconciliation.orphans.start | run started |
reconciliation.orphans.mark | item marked orphaned (or would be, dry-run) |
reconciliation.orphans.skip | item left untouched, lookup failed |
reconciliation.orphans.fail | item was orphaned but the write failed |
reconciliation.orphans.end | run totals |
reconciliation.shallow.start | run started |
reconciliation.shallow.grant | grant written to a path (or would be, dry-run) |
reconciliation.shallow.remove | redundant share removed (or would be, dry-run) |
reconciliation.shallow.skip | share left untouched, a lookup failed |
reconciliation.shallow.skipspace | space left untouched, its shares changed |
reconciliation.shallow.fail | a grant or a removal was needed but failed |
reconciliation.shallow.end | run totals |
reconciliation.expired.start | run started |
reconciliation.expired.remove | ACL of expired share removed (or would be, dry-run) |
reconciliation.expired.fail | cleanup failed, row kept for the next run |
reconciliation.expired.end | run totals |
Every line also carries job and run, a uuid identifying the run, so the two
logs stay readable if both jobs are pointed at the same file.
reconciliation.orphans.mark carries kind (share or publiclink), id,
reason, storage_id, opaque_id, owner, share_with, dry_run. Revert
with kind and id.
reconciliation.shallow.grant carries share, action (add or update),
path, storage_id, opaque_id, grantee, grantee_type, observed,
expected, dry_run. Revert with grantee and observed; an empty observed
means the entry was added and has to be removed.
reconciliation.shallow.remove carries share, path, storage_id,
opaque_id, grantee, grantee_type, level, reason, ancestor,
ancestor_path, ancestor_role, dry_run. Revert by clearing deleted_at on
the row with id share.
reconciliation.shallow.skipspace carries skipped_space, new_share (the
share that appeared, absent when a listing failed), grants and removals, the
number of changes held back, and dry_run. The two counts are absent when the
space could not be listed in the first place: the run never got as far as
deciding. Nothing was applied, so there is nothing to revert. Run the job on that
space again to apply them.
reconciliation.expired.remove carries share, storage_id, opaque_id,
grantee, grantee_type, level, dry_run. Revert by clearing expired_at
on the row with id share and adding back a level grant for grantee on the
resource.
Configuration
| Key | Type | Default | Description |
|---|---|---|---|
schedule | string | - | Schedule is the interval the job runs on, e.g. “@daily”. source |
dry_run | bool | - | DryRun, when set, makes the job log and report what it would do without touching the share database. source |
run_on_start | bool | - | RunOnStart fires the job once as soon as the runner starts, instead of waiting a full interval for the first run. source |
log_file | string | - | LogFile is the path the job writes its own log to. It takes “stdout” or “stderr” to write to the standard streams instead. source |
spaces_per_batch | int | - | SpacesPerBatch is how many spaces one listing covers, trading memory for round trips. The bound is in spaces, not in shares, so one space holding a great many shares is still read whole. Unset means no batching: every space in one listing. source |