 |
Project
|
|
Project
|
This is a static utilities class with the setters and getters of the various fields of the o2::headers::RAWDataHeaderVXXX classes of different versions. They allow uniform set/get access either via RDH reference of its pointer irrespective of the header version (starting from V3). Additionally, it uses a generic o2::header::RDHAny which can be morphed to any real RDH version (and vice versa). The sample of the code which can access the RDH in a version-idependent way is:
Utility class for interaction record -> HBF conversion and sampling of IRs for which the HBF RDH should be added to the raw data from the CRU. Should be used for MC->raw conversion to ensure that:
payloadIR (InteractionRecord = orbit/bc) for which payload is being created, dst vector with the IR's (between fromIR and payLoadIR) for which the HBFs should be generated and flushed to raw data before filling the payload of payLoadIR.Extra data member HBFUtil.maxNOrbits (= highest possible orbit by default) is used only in the RawFileWriter as a maximum orbit number (with respect to 1st orbit) for which detector payload is considered.
See Detectors/Raw/test/testHBFUtils.cxx for details of usage of this class (also check the jira ticket about the MC->Raw conversion).
A class to facilitate writing to files the MC data converted to raw data payload. Payload will be formatted according to CRU data specifications and should be simular to produced by the o2-readout-exe (and can be pumped into the DPL workflows)
The detector code for MC to raw conversion should instatiate the RawFileWriter object and:
cruID, linkID, CRU endPointID and feeID or an RDH containing similar infomation. It also can provide a file name to write the data of this link. Multiple links may write their data to the same file: By default the writer will be created for CRU detector (full constructor is RawFileWriter(o2::header::DataOrigin origin = o2::header::gDataOriginInvalid, bool cru=true)). In order to create it for RORC detector use e.g. RawFileWriter writer { "EMC", false }. It will automatically impose triggered readout mode and (i) every trigger will be written with its own RDH; (ii) no RDH.stop will be added in the end of the trigger; (iii) the start of the N-th TimeFrame will be assumed implicitly at the first trigger_orbit >= SOX_orbit + N*HBFUtils.orbitFirst, where the SOX_orbit is the orbit where SOX flag was set (start of continuous or start of triggered mode is set in the beginning with the orbit/bc corresponding to HBFUtils::getFirstIR()).
If needed, user may set manually the non-mutable (for given link) fields of the link RAWDataHeader via direct access to lnkC.rdhCopy. This fields will be cloned to all RDHs written for this link.
RawFileWriter, providing the link information and the o2::InteractionRecord corresponding to payload. The data must be provided as a gsl::span<char>and contain only detector GBT (16 bytes words) payload. The annotation by RDH, formatting to CRU pages and eventual splitting of the large payload to multiple pages will be done by theRawFileWriter. @icode{cpp} writer.addData(cru_0, link_0, endpoint_0, {bc, orbit}, gsl::span( (char*)payload_0, payload_0_size), <optional arguments> ); ... o2::InteractionRecord ir{bc, orbit}; writer.addData(rdh, ir, gsl::span( (char*)payload_f, payload_f_size ), <optional arguments>); @endicode where <optional arguments> are currently:bool preformatted = false, uint32_t trigger = 0, uint32_t detField = 0` (see below for the meaning).By default the data will be written using o2::header::RAWDataHeader. User can request particular RDH version via writer.useRDHVersion(v).
Note that the addData will have no effect if its {bc, orbit} exceeds the HBFUtils.getFirstIR() by more than HBFUtils.maxNOrbits.
The RawFileWriter will take care of writing created CRU data to file in super-pages whose size can be set using writer.setSuperPageSize(size_in_bytes) (default in 1 MB).
The link buffers will be flushed and the files will be closed by the destor of the RawFileWriter, but this action can be also triggered by write.close().
In case detector link payload for given HBF exceeds the maximum CRU page size of 8KB (including the RDH added by the writer; this may happen even if it the payload size is less than 8KB, since it might be added to already partially populated CRU page of the same HBF) it will write on the page only part of the payload and carry over the rest on the extra page(s). By default the RawFileWriter will simply chunk payload as it considers necessary, but some detectors want their CRU pages to be self-consistent: they add to their payload detector-specific trailer and header in the beginning and in the end of the CRU page. In case part of the data is carried over to another page, this extra information need to be repeated (possibly, with some modifications). In orger to customize payload splitting, detector code may provide implement and provide to the RawFileWriter a callback function with the signature:
The role of this method is to suggest to the writer how to split the payload: if it was provided to the RawFileWriter using writer.setCarryOverCallBack(pointer_on_the_converter_class); then the RawFileWriter will call it before splitting.
It provides to the converter carryOverMethod method the following info:
The method must return actual size of the bloc which can be written (<=maxSize). If this method populates the trailer, it must ensure that it returns the actual size such that actualSize + trailer.size() <= maxSize. In case returned actualSize is 0, current CRU page will be closed just with user trailer added (if any) and new query of this method will be done on the new CRU page.
By default, the carry-over callback is not called if remaining data fits to the free space of the 8KB page (or the super-page). In case the splitting affects the information written in the payload trailer, user may set writer.setApplyCarryOverToLastPage(true). With this flag set to ON, if there was at least one splitting for the user payload provided to addData method, then the carry-over method will be called also for the very last chunk (which by definition does not need splitting) and the supplied trailer will overwrite the tail of the this chunk instead of adding it incrementally.
Additionally, in case detector wants to add some information between empty HBF packet's opening and closing RDHs (they will be added automatically using the HBFUtils functionality for all HBFs w/o data between the orbit 0 and last orbit of the TF seen in the simulations), it may implement a callback method
If such method was registered using writer.setEmptyPageCallBack(pointer_on_the_converter_class), then for every empty HBF the writer will call it with
The data toAdd will be inserted between the star/stop RDHs of the empty HBF.
Adding empty HBF pages for HB's w/o data can be avoided by setting writer.setDontFillEmptyHBF(true) before starting conversion. Note that the empty HBFs still will be added for HBs which are supposed to open a new TF.
Some detectors (ITS/MFT) write a special header word after the RDH of every new CRU page (actually, different GBT words for pages w/o and with RDH.stop) in non-empty HBFs. This can be achieved by another call back method
It proveds the RDH of the page for which it is called, information if the previous had some payload and buffer to be filled by the used algorithm.
Some detectors signal the end of the HBF by adding an empty CRU page containing just a header with RDH.stop=1 while others may simply set the RDH.stop=1 in the last CRU page of the HBF (it may appear to be also the 1st and the only page of the HBF and may or may not contain the payload). This behaviour is steered by writer.setAddSeparateHBFStopPage(bool v) switch. By default end of the HBF is signaled on separate page.
RawFileWriter will check for every bc/orbit for which at least 1 link was filled by the detector if all other registered links also got addData call. If not, it will be enforced internally with 0 payload and trigger and detField RDH fields provided in the 1st addData call of this IR. This may be pose a problem in case detector needs to insert some header even for 0-size payloads. In this case it is detector's responsibility to make sure that all links receive their proper addData call for every IR.
Extra arguments:
preformatted (default: false): The behaviour described above can be modified by providing an extra argument in the addData method In this case provided span is interpretted as a fully formatted CRU page payload (i.e. it lacks the RDH which will be added by the writer) of the maximum size 8192-sizeof(RDH) = 8128 bytes. The writer will create a new CRU page with provided payload equipping it with the proper RDH: copying already stored RDH of the current HBF, if the interaction record ir belongs to the same HBF or generating new RDH for new HBF otherwise (and filling all missing HBFs in-between). In case the payload size exceeds maximum, an exception will be thrown w/o any attempt to split the page.
trigger (default 0): optionally detector may provide a trigger word for this payload, this word will be OR-ed with the current RDH.triggerTypedetField (default 0): optionally detector may provide a custom 32-bit word which will be assigned to RDH.detectorField.For further details see ITSMFT/common/simulation/MC2RawEncoder class and the macro Detectors/ITSMFT/ITS/macros/test/run_digi2rawVarPage_its.C to steer the MC to raw data conversion.
If the HBFUtils.obligatorySOR==false (default) the MC->Raw converted data will start from the 1st TF containing data (i.e. corresponding to HBFUtils.firstOrbitSampled), the SOX in the RDH will be set only if this TF coincides with the 1st TF of the Run (defined by the HBFUtils.orbitFirst). With HBFUtils.obligatorySOR==true old behaviour will be preserved: the raw data will start from TF with HBFUtils.orbitFirst with SOX always set and for CRU detectors all HBFs/TFs between HBFUtils.orbitFirst and 1st non-empty HBF will be filled by dummy RDHs.
With the introduction of RDHv7 some detectors write their payload w/o padding their GBT words to 16 bytes. Usually such detectors are required to align their CRU page size to certain size (see discussion). To delegate this padding for alignment reason to the RawFileWriter one should use methods:
A class for parsing raw data file(s) with "variable-size" CRU format. For every encountered link the DPL SubSpecification is assigned according to the formula used by DataDistribution:
This SubSpecification is used to define the DPL InputSpecs (and match the OutputSpecs). An exception will be thrown if 2 different link (i.e. with different RDH.feeId) of the same detector will be found to have the same SubSpec (i.e. the same cruID, linkID and PCIe EndPoint). Also, an exception is thrown if the block expected to be a RDH fails to be recognized as RDH.
TEMPORARY UPDATE: given that some detectors (e.g. TOF, TPC) at the moment cannot guarantee a unique (within a given run) mapping FEEID <-> {cruID, linkID, EndPoint}, the SubSpecification defined by the DataDistribution may mix data of different links (FEEDs) to single DPL input. To avoid this, the SubSpecification definition is temporarily changed to a 32 bit hash code (Fletcher32). When the new RDHv6 (with source ID) will be put in production, both DataDistribution and RawFileReader will be able to assign the SubSpecification in a consistent way, as a combination of FEEID and SourceID.
Since the raw data (RDH) does not contain any explicit information equivalent to o2::header::DataOrigin and o2::header::DataDescription needed to create a valid DPL OutputSpec, for every raw data file provided to the RawFileReader the user can attach either explicitly needed origin and/or description, or to use default ones (which also can be modified by the user). By default, RawFileReader assumes o2::header::gDataOriginFLP and o2::header::gDataDescriptionRawData (leading to FLP/RAWDATA OutputSpec).
The typical initialization of the RawFileReader is:
Note, that alternatively the input to the reader may be provided via text configuration file as:
The input configuration must have following format (parsed using Common-O2::ConfigFile):
The typical loop to read the data from already initialized reader is:
Note: every input file may contain data from different CRUs or GBT links, but mixing of different detectors specifications (defined by o2::header::DataOrigin and o2::header::DataDescription) is not allowed. Data from the same detector may be split to multiple files link-wise and/or time-wise. In the latter case the input files must be provided ordered in time (HBF orbit in the RDH).
The workflow takes an input from the configuration file (as described in RawFileReader section), reads the data and sends them as DPL messages with the OutputSpecs indicated in the configuration file (or defaults). Each link data gets SubSpecification according to DataDistribution scheme.
If --loop argument is provided, data will be re-played in loop. The delay (in seconds) can be added between sensding of consecutive TFs to avoid pile-up of TFs. By default at each iteration the data will be again read from the disk. Using --cache-data option one can force caching the data to memory during the 1st reading, this avoiding disk I/O for following iterations, but this option should be used with care as it will eventually create a memory copy of all TFs to read.
At every invocation of the device processing callback a full TimeFrame for every link will be added as a multi-part FairMQ message and relayed by the relevant channel. By default each HBF will start a new part in the multipart message. This behaviour can be changed by providing part-per-sp option, in which case there will be one part per superpage (Note that this is incompatible to the DPLRawSequencer).
By the default the DataProcessingHeader of each message will have its creation time set to now(). This can be changed by passing an option --configKeyValues "HBFUtils.startTime=<t>" with t being desired run start time in milliseconds: in this case the creation time will be defined as t + (firstTForbit-HBFUtils.orbitFirst)*orbit_duration in milliseconds.
The standard use case of this workflow is to provide the input for other worfklows using the piping, e.g.
Option --raw-channel-config <confstring> forces the reader to send all data (singlefair::mq::Partscontaining the whole TF) to raw FairMQ channel, emulating the messages from the DataDistribution. To inject such a data to DPL one should use a parallel process starting witho2-dpl-raw-proxy. An example (note–session default` added to every executable):
For testing reason one can request dropping of some fraction of the TFs for particular detector. With option --drop-tf "ITS,3,2;TPC,10" the reader will not send the output for ITS in TFs with (TFid%3)==2 and for TPC in TFs with (TFid%10)==0. Note that in order to acknowledge the TF, the message {FLP/DISTSUBTIMEFRAME/0} will still be sent even if all detector's data was dropped for a given TF.
Allows to check the correctness of CRU data (real or simulated) stored in the binary file. Multiple files can be checked, with each file containing data for the same or distinct group of links.
Apart from the eventual exception produced for unrecognizable RDH or 2 links with the same cruID, linkID and PCIe EndPoint but different feeId (see RawFileReader), the following errors (check can be disabled by corresponding option) are reported (as ERROR) for every GBT link while scanning each file (the error counter of each link is incremented for any of this errors):
RDH.packetCounter) is not incremented by 1 (--nocheck-packet-increment to disable check)RDH.pageCnt) is not incremented by 1 (--nocheck-page-increment to disable)RDH.stop is set at the very 1st page of the HBF (--nocheck-stop-on-page0 to disable)RDH.stop = 1 received (--nocheck-missing-stop to disable)--nocheck-starts-with-tf to disable)--nocheck-hbf-per-tf to disable)--nocheck-hbf-jump to disable)--nocheck-no-spage-for-tf to disable)After scanning each file, for every link the largest SuperPage size (as the size of the largest contiguos block containing data of this link). A warning is printed if this size exceeds the nominal SuperPage size (D=1MB, can be imposed by user).
For the correct data the output should look like this:
o2-raw-tf-reader-workflow allows to inject into the DPL the (s)TF data produced by the DataDistribution, avoiding running DD sTFReader in a separate terminal and sending data via proxies. The relevant command line options are:
input data (obligatory): comma-separated list of input data files and/or files with list of data files and/or directories containing files
max TF ID to process (<= 0 : infinite)
This is a tf-reader device local option allowing selective reading of particular TFs. It is useful when dealing with TF files containing multiple TFs. The comma-separated list of increasing TFs indices must be provided in the format parsed by the RangeTokenizer<int>, e.g. 1,4-6,.... Note that the index corresponds not to DataHeader.TFcounter of the TF but to the reader own counter incremented throught all input files (e.g. if 10 raw-TF files with 20 TFs each are provided for the input and the selection of TFs 0,2,22,66 is provided, the reader will inject to the DPL the TFs at entries 0 and 2 from the 1st raw-TF file, entry 5 of the second file, entry 6 of the 3d and will finish the job.
loop N times (-1 = infinite) over input files (but max-tf has priority if positive)
delay in seconds between consecutive TFs sending (provided TFs can be read fast enough)
copy command for remote files
regex string to identify TF files: optional to filter data files (if the input contains directories, it will be used to avoid picking non-TF files)
regex string to identify remote files (they will be copied to /tmp/<random>/ directory, then removed after processing)
max TFs to cache in memory: builds TFs asynchrously to sending, may speed up processing but needs more shmem.
max TF files queued (copied for remote source). For local files almost irrelevant, for remote ones asynchronously creates local copy.
verbosity level (1 or 2: check RDH, print DH/DPH for 1st or all slices, >2 print RDH)
optional raw FMQ channel for non-DPL output, similar to o2-raw-file-reader-workflow.
Since a priory it is not known what kind of data is in the raw TF file provided on the input, by default the reader will define outputs for the raw data of all known detectors (as subspecification-wildcarded <DET>/RAWDATA). Additionally, since some detectors might have done processing on the FLP (in which case the DD TF will contain derived data, e.g. cells), the reader will open extra outputs for all kinds of messages for which we know they can be produced on the FLP. Following 3 partially redundant options allow to decrease the number of defined outputs:
list of detectors to select from available in the data, outputs for others will not be created, their data will be skipped.
list of detectors for which non-raw outputs (if any) are discarded.
list of detectors for which raw outputs are discarded.
The raw data will be propagated (if present) only if the detector is selected in --onlyDet and NOT selected in --non-raw-only-det. The non-raw data will be propagated (if defined for the given detector and present in the file) only if the detector is selected in --onlyDet and NOT selected in --raw-only-det.
To apply TF rate limiting (i.e. make sure that no more than N TFs are in processing) provide --timeframes-rate-limit <N> --timeframes-rate-limit-ipcid <IPCID> too all workflows (e.g. via ARGS_ALL). The IPCID is the NUMA domain ID (usually 0 on non-EPN workflow). Additionally, one may throttle on the free SHM by providing an option to the reader --timeframes-shm-limit <shm-size>.
The workflow o2-raw-tf-dump-workflow allows to convert rawTF files to raw files used by o2-raw-file-reader-workflow (or readout.exe replay), creating also raw-file-reader configuration files (one per detector). Example of usage:
rawStat.C: writes into the tree the size per HBF contained in the raw data provided in the RawFileReader config file. No check for synchronization between different links is done. 
1.9.8