@@ -64,20 +64,22 @@ Analyzing a traced build is a multi-stage process that involves:
6464 build,
6565
6666 - analyzing the build graph to determine the source to target relationships,
67- such as source code files being built into a binary,
67+ such as source code files being built into a binary,
6868
6969 - optionally creating graphical representations to visualize subset of
7070 your build graph.
7171
72- Each of these steps is performed by invoking tracecode from the command line
72+ Each of these steps is performed by invoking ` tracecode ` from the command line
7373with different options and arguments.
7474
7575
7676Run the trace analysis with::
77+
7778 tracecode <options> <command> <arguments>
7879
7980
8081For command help use::
82+
8183 tracecode -h
8284
8385
@@ -91,6 +93,7 @@ Tutorial
9193
92941. Parse the collected raw traces
9395---------------------------------
96+
9497Create a new empty directory to store parsed traces. Then parse using the
9598"parse" command::
9699
@@ -102,7 +105,8 @@ This will parse the traces and ensure they can be processed and are complete
102105
1031062. Collect the inventory of files processed during the tracing
104107--------------------------------------------------------------
105- If traces are complete the next step is to collect the inventories of files
108+
109+ If traces are consistent the next step is to collect the inventories of files
106110reads and writes. Use the "list" command (which should be called inventory).
107111It creates two files from a parsed trace: a list of files being only read and
108112a list of files being written::
@@ -115,22 +119,24 @@ The list command extracts all the paths used in the traces.
115119
1161203. optional but recommended: Filter your parsed traces
117121------------------------------------------------------
122+
118123The next step is to review these reads and writes and decide which ones could
119124be filtered out as they may not contribute interesting data to the build graph
120125and the analysis.
121126
122127This includes typically:
123- /etc/*
124- /proc/*
125- the build log files if any
126- Some standard things in /usr/* and similar
128+
129+ - /etc/*
130+ - /proc/*
131+ - the build log files if any
132+ - Some standard things in /usr/* and similar
127133
128134For this you build a list of reads to ignore and writes to ignore (usually
129135patterns or plain lists) you stuffs these two lists in a two files and use the
130- filter command to filter out these reads and writes
136+ filter command to filter out these reads and writes.
131137
132138Beware of not filtering too much: temp files in /tmp you want to keep certain
133- makedepend (.po, etc) files you may not care for
139+ makedepend (.po, etc) files you may not care for.
134140
135141When you filter at first filter to a new directory so taht you do not replace
136142the original full parsed traces yet, so you can get comfy and refine your
@@ -141,23 +147,25 @@ out or prune from the trace Either a full path as found in the reads or writes
141147list, or a pattern as in /etc/* in which case everything matching /etc/* would
142148be filtered out like when you use glob patterns on the command line Use oe
143149path or pattern per line in a file. Note that it can be a single column csv
144- alright too
150+ alright too.
145151
146152
1471534. optional: Guess sources and targets
148154-------------------------------------
155+
149156You can use the "guess" command to guess sources and targets, but that is just
150157a guess. Guessing works ok on small well defined simple codebases, but might
151- noy likely be good on larger ones
158+ noy likely be good on larger ones.
152159
153160The guess goes this way:
154- * files that are only ever read from are likely the source/devel
155- * files that are only ever written to read are likely the target/deployed
161+ - files that are only ever read from are likely the source/devel
162+ - files that are only ever written to read are likely the target/deployed
156163
157164
158165
1591665. Assemble the inventory of sources an targets
160167-----------------------------------------------
168+
161169Once you have filtered your parsed trace, you need to create a list of files
162170that are your sources, origin development files and another list that are your
163171targets, deployed files. You need to build theses inventories each in a
@@ -174,12 +182,14 @@ devel/sources files and what is the list of deployed/targets files.
174182
1751836. Analyze sources to targets transformations
176184---------------------------------------------
185+
177186Then you can run either the analyze command to get the source to target
178187deployment analysis.
179188
180189
1811907. optional: Graph select subset of sources to targets transformations
182191----------------------------------------------------------------------
192+
183193You can selectively create a graphic tracing the transformation from several
184194sources to a one target or several targets to one sources with graphics
185195(selectively because this takes long time to run and large graphics are
@@ -190,8 +200,9 @@ impossible to visualize)
190200FAQ:
191201----
192202
193- Q: When parsing raw traces I am getting this error:
194- ERROR:tracecode: INCOMPLETE TRACE, 149249 orphaned trace(s) detected. First pid is: 3145728.
203+ Q: When parsing raw traces I am getting this error::
204+
205+ ERROR:tracecode:INCOMPLETE TRACE, 149249 orphaned trace(s) detected. First pid is: 3145728.
195206
196207A: This is a serious error and means that your trace is not coherent as some
197208process traces could not be related to the initial command launch graph and
@@ -200,8 +211,9 @@ commands and store the strace output in the same directory. You need to
200211recollect your traces starting with a clean empty directory.
201212
202213
203- Q: When parsing raw traces I am getting several warnings:
204- WARNING:tracecode: parse_line: Unable to decode descriptor for pid: 3097012, line: '1399882436.807573 dup2(5</extra/linux-2.6.32/scripts/mksysmap>, 255) = 255\n '
214+ Q: When parsing raw traces I am getting several warnings::
215+
216+ WARNING:tracecode:parse_line: Unable to decode descriptor for pid: 3097012, line: '1399882436.807573 dup2(5</extra/linux-2.6.32/scripts/mksysmap>, 255) = 255\n'
205217
206218A: This is just a warning that you can ignore most of the times. Here a file
207219descriptor 255 does not (and cannot) exist, hence the warning.
@@ -210,41 +222,40 @@ descriptor 255 does not (and cannot) exist, hence the warning.
210222Credits and related tools
211223-------------------------
212224
213- This implementation of an strace-based build tracer is
214- essentially an implementation of this papers:
225+ This implementation of an strace-based build tracer is essentially an implementation
226+ of this papers:
215227
216- Sander van der Burg published this article:
217- http://sandervanderburg.blogspot.be/2012/04/dynamic-analysis-of-build-processes-to.html
228+ Sander van der Burg published an article and paper:
218229
219- ... about this paper:
220- http://www.st.ewi.tudelft.nl/~sander/index.php/publications
230+ - http://sandervanderburg.blogspot.be/2012/04/dynamic-analysis-of-build-processes-to.html
231+ - "Discovering Software License Constraints: Identifying a Binary's Sources by Tracing Build Processes"
232+ - http://www.st.ewi.tudelft.nl/~sander/pdf/publications/TUD-SERG-2012-010.pdf
233+ - By Sander van der Burg, Julius Davies, Eelco Dolstra, Daniel M. German, Armijn Hemel.
234+ Technical Report TUD-SERG-2012-010, Software Engineering Research Group, Delft, The Netherlands, April 2012.
221235
222- "Discovering Software License Constraints:
223- Identifying a Binary's Sources by Tracing Build Processes"
224- By Sander van der Burg, Julius Davies, Eelco Dolstra,
225- Daniel M. German, Armijn Hemel.
226- Technical Report TUD-SERG-2012-010, Software Engineering Research Group, Delft, The Netherlands, April 2012.
227- http://www.st.ewi.tudelft.nl/~sander/pdf/publications/TUD-SERG-2012-010.pdf
228236
229- Later this similar paper relates a similar approach too :
230- http://web.archive.org/web/20160329060541/http://shanemcintosh.org/assets/ase2014_vanderburg.pdf
231- "Tracing Software Build Processes to Uncover License Compliance Inconsistencies"
232- By Sander van der Burg, Eelco Dolstra, Shane McIntosh, Julius Davies
233- Daniel M. German, and Armijn Hemel
237+ Later, this similar paper relates the same approach:
238+
239+ - "Tracing Software Build Processes to Uncover License Compliance Inconsistencies"
240+ - http://web.archive.org/web/20160329060541/http://shanemcintosh.org/assets/ase2014_vanderburg.pdf
241+ - By Sander van der Burg, Eelco Dolstra, Shane McIntosh, Julius Davies, Daniel M. German, and Armijn Hemel
234242
235243
236244The Chromium test team built "swarming.client", a test isolation
237245tools that was also a big inspiration for this tool too:
238- https://www.chromium.org/developers/testing/isolated-testing/infrastructure
239- https://chromium.googlesource.com/external/swarming.client/
240246
247+ - https://www.chromium.org/developers/testing/isolated-testing/infrastructure
248+ - https://chromium.googlesource.com/external/swarming.client/
249+
250+
251+ - memoize.py and fabricate use strace to track file dependencies
252+ using a similar approach to this tool.
253+ - https://github.com/kgaughan/memoize.py
254+ - https://code.google.com/archive/p/fabricate/
241255
242- memoize.py and fabricate use strace to track file dependencies
243- using a similar approach to this tool. And this article provides some good background on that topic:
256+ - And this article provides some good background on the same topic:
244257https://news.ycombinator.com/item?id=9356433 :
245258
246- - https://github.com/kgaughan/memoize.py
247- - https://code.google.com/archive/p/fabricate/
248259
249260
250261http://buildaudit.sourceforge.net/ is a related build tracing tool that
@@ -253,7 +264,7 @@ handles ptrace directly ass opposed to rely on strace for tracing.
253264
254265Electric cloud is tool has some ways to track which files are accessed during a build
255266using ptrace of LD_PRELOAD (or a custom file system)
256- - https://electric-cloud.com/
267+ - https://electric-cloud.com/
257268
258269
259270License
0 commit comments