reconciliation

Reconciles the share database against the state of the storage.

Package
github.com/cs3org/reva/v3/pkg/reconciliation
Source
pkg/reconciliation · Go reference
Used by
reconciliation

Overview

Package reconciliation reconciles the share database against the state of the storage. It holds two jobs: orphan detection, which marks the shares and public links whose resource or recipient is gone, and the shallow check, which repairs the ACLs on the paths that carry a share. The engine is storage-driver agnostic: it reads shares from the database and resolves resources, identities and providers through the gateway (CS3).

Every line a job logs carries an “event” naming what happened, so a run can be replayed or reverted by filtering on it rather than by parsing free-form messages. An event is the job’s own name followed by the step: every job has start, skip, fail and end, plus one event for the change it makes. Each line also carries “job” and “run”, the uuid of the run it belongs to, so jobs stay apart even when they log to the same file.

The jobs span the shares and the public links, so they are wired as their own serverless service rather than by one of the grpc share services. They go through a share manager and a public share manager, named by driver like everywhere else. Marking an item orphaned is not part of the CS3 API, so a driver that does not implement it is refused at startup; today only sql does, which is why it is the default for both.

Each job is enabled and configured on its own, so one can be left in dry-run, or rescheduled, or pointed at another log, without touching the other.

[serverless.services.reconciliation]
jobs = ["orphan", "shallow", "expired"]
share_driver       = "sql"
publicshare_driver = "sql"
service_user_name  = "cboxreco"
service_user_uid   = 12345
service_user_gid   = 2766

[serverless.services.reconciliation.orphan]
schedule = "@daily"      # "@every <dur>" | "@hourly" | "@daily" | "@weekly"
dry_run  = false
log_file = "/var/log/revad/reconciliation-orphan.log"
spaces_per_batch = 500   # omit to read everything in one go

[serverless.services.reconciliation.shallow]
schedule = "never"       # or a cadence, on-demand runs work either way
dry_run  = true
log_file = "/var/log/revad/reconciliation-shallow.log"
spaces_per_batch = 500   # omit to read everything in one go

[serverless.services.reconciliation.expired]
schedule = "@hourly"
dry_run  = false
log_file = "/var/log/revad/reconciliation-expired.log"

A job runs only if it is listed in jobs. orphan and expired need a schedule; on shallow it is optional and also takes never, since that job can be triggered on demand whether or not it has a cadence. log_file defaults to the path shown above for each, and both drivers default to sql, so all four keys above the service user can be omitted.

spaces_per_batch limits one batch to a given number of spaces. This has been introduces to avoid memory spikes due to large numbers of shares being loaded into memory at the same time. If no value is set, all shares will be loaded into memory at the same time.

With a batch size, a row whose space_id is NULL is not visited: there is no space to put it in a batch under. The hierarchy check the share API runs already skips such a share for the same reason.

The drivers take their configuration from [serverless.services.reconciliation.share_drivers.<name>] and [serverless.services.reconciliation.publicshare_drivers.<name>], the same keys the usershareprovider and the publicshareprovider pass. The sql driver reads its database settings from [shared], so neither section is normally needed, and neither is jwt_secret.

Identity

The jobs runner hands a run a bare context, but we need a valid auth so we can do stat’s etc. Therefore, each run mints itself a token for service_user_name and sends it with every call. The three service_user_* keys are required and the account has to be a real one: EOS reads the ACLs of a node as the caller before handing them out, and its driver refuses a caller whose uid or gid is zero, so root does not work (for CERNBox, cbox does the trick). If skip_user_groups_in_token is set, the account also has to resolve in the user provider, since the auth interceptor looks its groups up.

Jobs

reconciliation.orphans marks the shares and public links whose resource or recipient no longer exists. With a spaces_per_batch it asks both stores which spaces they hold an item in and reads those spaces a batch at a time. Each item is judged on its own, so a batch is a listing and nothing more. If a batch cannot be listed the run fails there: the job is idempotent, so what it marked stands and the next run picks up the rest.

reconciliation.shallow writes the ACL entry each share implies onto its path, when the storage has no entry or the wrong permissions. On the storage it only adds and corrects entries. It never removes an ACL entry. In the share database it does remove rows: a share that a higher share of the same recipient makes redundant is deleted (soft). See below for the details.

reconciliation.expired removes the ACL entries of the shares whose expiration date has passed. The share manager hides an expired share from the recipient, but its ACL entry stays until this job runs. For each expired share it removes the entry, reapplies the entries of the same recipient’s live shares above and below it (as RemoveShare does), and sets expired_at on the row. The row is not deleted, so the owner still sees the share. A share with expired_at set is not picked up again. If a step fails, expired_at stays empty and the next run tries again.

Start a run with reva admin jobs run reconciliation.shallow [space=<space-id>]. Leave the space out to cover every space. This needs the admin API enabled. The command prints a run id: reva admin jobs status <run-id> shows the state and the totals, reva admin jobs cancel <run-id> stops the run.

With a schedule, the job also runs on a cadence over every space. That run has its own name, so reva admin jobs trigger reconciliation.shallow.scheduled starts it before its next fire.

With a spaces_per_batch, a full run asks the database which spaces hold a share and reads them a batch at a time. It decides and applies one space at a time either way, so a run scoped with space=<space-id> is that same work over one space.

A run works out the changes of a space before it writes any of them, and a user can share in the meantime. So the shares of a space are listed again right before its changes go out. A space that gained a share since the run listed it is left untouched, whole, and picked up by the next run: the new share is not in what the run compared, and it can be what makes a share the run wants to remove no longer redundant. Only the spaces the run has something to change in are looked at again. Their ids come back in the run’s totals under skipped_spaces, so a retry can be scoped to exactly the spaces that were held back.

A share below another share of the same recipient is redundant when it grants the same access (covered in the totals) or less (conflicting). The job deletes the row of such a share, which is what the share API does when the higher share is created. The delete is soft, so you can undo it in the database. The ACL entry on the path stays.

A share whose expiration date has passed is left out (expired in the totals). The job does not write its entry, and it does not make a share below it redundant. The share API ignores expired shares in the same way.

Logs

Each job writes its own log (well, in fact, more like a journal), separate from revad’s. This is always written in JSON so that it is easy to parse by any tool in case we need to revert certain actions.

An event is the job’s own name followed by the step, so one job’s lines never have to be told apart from the other’s by hand.

eventmeaning
reconciliation.orphans.startrun started
reconciliation.orphans.markitem marked orphaned (or would be, dry-run)
reconciliation.orphans.skipitem left untouched, lookup failed
reconciliation.orphans.failitem was orphaned but the write failed
reconciliation.orphans.endrun totals
reconciliation.shallow.startrun started
reconciliation.shallow.grantgrant written to a path (or would be, dry-run)
reconciliation.shallow.removeredundant share removed (or would be, dry-run)
reconciliation.shallow.skipshare left untouched, a lookup failed
reconciliation.shallow.skipspacespace left untouched, its shares changed
reconciliation.shallow.faila grant or a removal was needed but failed
reconciliation.shallow.endrun totals
reconciliation.expired.startrun started
reconciliation.expired.removeACL of expired share removed (or would be, dry-run)
reconciliation.expired.failcleanup failed, row kept for the next run
reconciliation.expired.endrun totals

Every line also carries job and run, a uuid identifying the run, so the two logs stay readable if both jobs are pointed at the same file.

reconciliation.orphans.mark carries kind (share or publiclink), id, reason, storage_id, opaque_id, owner, share_with, dry_run. Revert with kind and id.

reconciliation.shallow.grant carries share, action (add or update), path, storage_id, opaque_id, grantee, grantee_type, observed, expected, dry_run. Revert with grantee and observed; an empty observed means the entry was added and has to be removed.

reconciliation.shallow.remove carries share, path, storage_id, opaque_id, grantee, grantee_type, level, reason, ancestor, ancestor_path, ancestor_role, dry_run. Revert by clearing deleted_at on the row with id share.

reconciliation.shallow.skipspace carries skipped_space, new_share (the share that appeared, absent when a listing failed), grants and removals, the number of changes held back, and dry_run. The two counts are absent when the space could not be listed in the first place: the run never got as far as deciding. Nothing was applied, so there is nothing to revert. Run the job on that space again to apply them.

reconciliation.expired.remove carries share, storage_id, opaque_id, grantee, grantee_type, level, dry_run. Revert by clearing expired_at on the row with id share and adding back a level grant for grantee on the resource.

Configuration

KeyTypeDefaultDescription
schedulestring-Schedule is the interval the job runs on, e.g. “@daily”. source
dry_runbool-DryRun, when set, makes the job log and report what it would do without touching the share database. source
run_on_startbool-RunOnStart fires the job once as soon as the runner starts, instead of waiting a full interval for the first run. source
log_filestring-LogFile is the path the job writes its own log to. It takes “stdout” or “stderr” to write to the standard streams instead. source
spaces_per_batchint-SpacesPerBatch is how many spaces one listing covers, trading memory for round trips. The bound is in spaces, not in shares, so one space holding a great many shares is still read whole. Unset means no batching: every space in one listing. source
Generated from pkg/reconciliation. Edit its package doc comment or README.md to change this page.