Hi, (removing Joerg to avoid spamming his inbox during holidays...) On Thu, 27 Oct 2011, Krisztian Krajczar wrote: > the two Pede jobs have successfully finished. > The original Pede job (no modification of weights) have consumed ~31 GB > memory: > > Memory space: total 32.000000 GB > used 31.085153 GB = 97.14 % > > This remained true for the second (weighted cosmics) Pede job: > > Memory space: total 32.000000 GB > used 31.085153 GB = 97.14 % As expected: no new off-diagonal entries can appear by weighting... > The weighting actually worked, since the pede dump for the second job also > contains this line: > !!!!! 50 weighted binary files Good. I suggest to run another pede job, down-weighting the cosmics by 0.5, then setup a new pede directory to start from ideal geometry and repeat the three pede jobs. Then we will have enough food for the Z-validation, to see what effect the different scenarios have on eta/phi modes. (Following the discussion from PVT yesterday, we should certainly look at delta eta as well...) Waiting for the pede jobs to finish one could work a bit on how to make the track input more similar for data and MC: data minbias vs MC minbias sum of MC isolated muons vs data isolated muons data peak cosmics (CRAFT + interfill) vs mc peak cosmics data deco cosmics (interfill [+cdc?]) vs mc deco cosmics From mp0763 directory one should be able to find out about the data samples... Cheers Gero >> I have submitted an other Pede job where I increased the weigths of the >> cosmics by 2 (both for deco and peak cosmics). >> >> I selected the .dat files to be reweighted in alignment_merge.py by >> checking the input datasets for the various Mille jobs in the >> jobData/jobXYZ/the.py files. >> >> The original Pede job is still running, the new one is pending: >> MMM jobm 181880464 RUN 0 148182 0 0.000 +0 >> MMM jobm1 181983624 PEND 0 0 0 0.000 >> >> The db files will appear in >> /afs/cern.ch/cms/CAF/CMSALCA/ALCA_TRACKERALIGN/MP/MPproduction/mp0896/jobData/jobm >> and >> /afs/cern.ch/cms/CAF/CMSALCA/ALCA_TRACKERALIGN/MP/MPproduction/mp0896/jobData/jobm1 >> >> Cheers, >> Krisztian >> >>> OK, a short summary, including some more docu: >>> >>> 1) Z-validation: Roberto will take care of that with the help of Natalie. >>> First object to test is >>> >>> /afs/cern.ch/cms/CAF/CMSALCA/ALCA_TRACKERALIGN/MP/MPproduction/mp0895/jobData/jobm/alignments_MP.db >>> >>> i.e. my test setup. Do not forget to use also the bow parameters in the >>> same file. >>> >>> (To enable other persons in future, a Wiki page will be prepared to tell >>> how to run the Z-validation...) >>> >>> 2) Alignment >>> Krisztian's first re-running of Gero's test setup should arrive in a few >>> hours in .../mp0896/jobData/jobm/alignments_MP.db. >>> Should be ~identical to Gero's. (Can someone run a geometry comparison??) >>> Next steps are to >>> - do a 2nd pede job on the same mille binaries, but weighting up the >>> cosmics with factor 2 >>> - a new setup starting from ideal alignment >>> >>> Caveat (not discussed in the meeting): >>> In MC we usually keep the BPIX in fixed, but the current setup is from >>> data where the TOB is kept in place - needs to be fixed (or simply BPIX >>> centred in the end..?) >>> >>> 3) Track based validation >>> First test by Adam with a similar setup as used recently - py-config file >>> to be checked by Roberto and me. >>> >>> To read in the bow values you have to add (all-in-one-tool does not take >>> care!!): >>> process.trackerBowedSensors = >>> CalibTracker.Configuration.Common.PoolDBESSource_cfi.poolDBESSource.clone( >>> connect = cms.string(''), >>> >>> toGet = cms.VPSet(cms.PSet(record = >>> cms.string('TrackerSurfaceDeformationRcd'), >>> tag = cms.string('Deformations') >>> ) >>> ) >>> ) >>> process.prefer_trackerBowedSensors = cms.ESPrefer("PoolDBESSource", >>> "trackerBowedSensors") >>> >>> where '' is to be replaced by >>> >>> sqlite_file:/afs/cern.ch/cms/CAF/CMSALCA/ALCA_TRACKERALIGN/MP/MPproduction/mp0763/jobData/jobm/alignments_MP.db'), >>> >>> for real data and by >>> >>> sqlite_file: >>> >>> for the MC alignments to be studied. >>> >>> I hope all is clear - we are aware that Monday and Tuesday are public >>> holidays in Hungary, so the aim is to have first validation from Adam >>> ready on Friday - and we aim for presenting first results next Thursday. >>> >>> Thanks for your efforts! >>> >>> Gero >>> >>> On Wed, 26 Oct 2011, Krisztian Krajczar wrote: >>> >>>> Hi Gero, >>>> >>>> I had successfully run the Mille jobs using a copy of your setup in >>>> mp0895. However, the Pede job is still running, since ~11am... I hope it >>>> will get completed soon. >>>> >>>> Cheers, >>>> Krisztian >>>> >>>>>> Hi Gyorgy, Krisztian and Adam, >>>>> >>>>> >>>>> >>>>> one correction/addition: >>>>> >>>>>> - take care that setup_align.pl is executable (chmod ...) >>>>> >>>>> Change the mass storage directory to point to the user's EOS directory: >>>>> Search for the line with the FIXME comment and then do e.g. >>>>> >>>>> my $mssDir = "/store/caf/user/krajczar/MPproduction/${mpsdirname}"; >>>>> >>>>> The directory part before ${mpsdirname} should be created with >>>>> >>>>> cmsMkdir /store/... >>>>> >>>>> BTW: >>>>> My job in mp0895 ran succesfully through. The memory margin was not >>>>> large (from jobData/jobm/pede.dump.gz): >>>>> >>>>> Memory space: total 32.000000 GB >>>>> used 31.085153 GB = 97.14 % >>>>> >>>>> But I have not investigated the output... >>>>> >>>>> Cheers >>>>> >>>>> Gero >>>>> >>>>>> - execute it, best is to store its output for reference: >>>>>> ./setup_align.pl > dump >>>>>> - send jobs using mps_fire.pl, check status with >>>>>> mps_stat.pl/mps_fetch.pl >>>>>> and finally send the ped job mps_fire.pl -m >>>>>> [or use mps_auto.pl 300, see >>>>>> https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideMillepedeIIProductionEnvironment] >>>>>> >>>>>> The pede job might not run out-of-the-box due to memory issues >>>>>> (mps_fetch.pl should tell you once the merge job is DONE). Please let >>>>>> me know the output to diagnose and suggest further actions. >>>>>> >>>>>> Input samples are currently >>>>>> - isolated muons from W (16.3 M tracks) >>>>>> - isolated muons from QCD-mu enriched events (2.3 M tracks, some [!] >>>>>> lower >>>>>> pt) >>>>>> - minbias tracks (450 k events - about 4.5 M tracks??) >>>>>> - Zmumu events using mass constraint (390 k events used in pede) >>>>>> - peak mode cosmics (1.8 M used tracks) >>>>>> - deco mode cosmics (1.7 M used tracks) >>>>>> >>>>>> This breakdown is experimental: >>>>>> The aim is to have similar overall statistics and pt/eta/phi >>>>>> distributions as Joerg has used in data (see his presentation last >>>>>> Thursday). >>>>>> You find some basic track distribution in the millePedeMonitor*root >>>>>> files that you find in the jobData/jobLMN directories (after the pede >>>>>> job is run you find it in only in EOS storage where the binaries are as >>>>>> well). >>>>>> A subtask is to get 'similar' distributions, by either changing the >>>>>> number of input events per job for a specific sample (e.g. search for >>>>>> maxEvents in alignment_minbias.py), or by weighting the samples in the >>>>>> pede fit. This can be done by weighting adding, e.g. for a weight of >>>>>> 0.8, "-- 0.8" behind the the .dat files in the config of the merge job, >>>>>> e.g. see >>>>>> .../mp0860/jobm_it2/alignment_merge.py. >>>>>> For this you have to find out which binary number belongs to which >>>>>> sample - the dump from setting up the aignment can help to diagnose >>>>>> this (note that the order can change due to the 'confhash' map used in >>>>>> perl...!) >>>>>> >>>>>> But before experimenting with/fine tuning of the input, do the >>>>>> validation - then we might have to iterate. >>>>>> >>>>>> 2) Z validation >>>>>> =============== >>>>>> The Z-mass peak position as a function of eta and phi of the >>>>>> positive/negative muon, maybe also something like vs phi for eta > 0.9 >>>>>> or so. Either write you own simple analyser (as Joerg did), otherwise I >>>>>> hope that Roberto Castello (in CC) can help. >>>>>> >>>>>> It would be good to overlay >>>>>> - ideal alignment >>>>>> - the current MC alignment (i.e. the start geometry in the current >>>>>> setup) >>>>>> - the data >>>>>> (- maybe Joerg's first shot) >>>>>> >>>>>> 3) Track based validation >>>>>> ========================= >>>>>> We need the DMR and normlaised DRR plots as youhave produced them in >>>>>> the past already. The tricky things are >>>>>> a) to get similar statistics and pt/eta distributions for the sample >>>>>> testing the new MC alignment with MC events and the data alignment >>>>>> with >>>>>> data >>>>>> b) not to forget to load the correct TrackerSurfaceDeformationRcd as >>>>>> determined in your alignment >>>>>> >>>>>> To achieve a) I suggest to validate in data on the longest of our >>>>>> periods (B-I), but do a pt cut of e.g. 20 GeV and then validate MC on a >>>>>> fraction of similar size from >>>>>> /WJetsToLNu_TuneZ2_7TeV-madgraph-tauola/Summer11-TkAlMuonIsolated-PU_S4_START42_V12-v1/ALCARECO >>>>>> >>>>>> >>>>>> =============================== >>>>>> So far for today - I guess many details are left out, but of course you >>>>>> have some experience with alignment and validation already, so I hope >>>>>> you can fill some holes. >>>>>> Now I just remind that validating is probably taking more time than >>>>>> actully doing the alignment, so I suggest to get used to the validation >>>>>> already testing the existing alignments (ideal, current misalignment >>>>>> scenarios). >>>>>> >>>>>> Cheers >>>>>> >>>>>> Gero >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> ----------------------------------------------------------------------- >>>>> Gero Flucke >>>>> - Analysis Centre, Helmholtz Alliance "Physics at the Terascale" >>>>> * Statistics Tools >>>>> - CMS: Tracker Alignment Convenor >>>>> DESY/CMS, Notkestr. 85, D-22607 Hamburg, Germany >>>>> Bldg. 1e, Rm. 02.501 >>>>> phone: +49 (0)40 8998 3525 >>>>> fax: +49 (0)40 8998 3092 >>>>> >>>> >>> >>> -- >>> ----------------------------------------------------------------------- >>> Gero Flucke >>> - Analysis Centre, Helmholtz Alliance "Physics at the Terascale" >>> * Statistics Tools >>> - CMS: Tracker Alignment Convenor >>> DESY/CMS, Notkestr. 85, D-22607 Hamburg, Germany >>> Bldg. 1e, Rm. 02.501 >>> phone: +49 (0)40 8998 3525 >>> fax: +49 (0)40 8998 3092 >>> >> > -- ----------------------------------------------------------------------- Gero Flucke - Analysis Centre, Helmholtz Alliance "Physics at the Terascale" * Statistics Tools - CMS: Tracker Alignment Convenor DESY/CMS, Notkestr. 85, D-22607 Hamburg, Germany Bldg. 1e, Rm. 02.501 phone: +49 (0)40 8998 3525 fax: +49 (0)40 8998 3092