Argo user´s manual, Seoul update comments for version

Argo user´s manual, Seoul update comments for version Comments on the proposed update of 1Argo user’s manual following Seoul meeting decisions Feedbacks on the "User's manual" update proposal sent on 13/02/2012 13/02/2012 Message from Thierry Carval: the manual update proposal is online for comments Dear Megan and Argo-dm, Thank you very much for this new version 2.4 of the trajectory file format. I only have a very minor remark; on page 5, use "code" instead of "flag" for measurement_code. I inserted your document in the draft of the next Argo user's manual. The draft is available from : Following Seoul's data-management meeting, this draft includes the following features, listed on page 8 in the "History of document" chapter : , ?2.3 Megan Scanderbeg : update of trajectory format following Seoul trajectory & ADMT12 meeting , ?3.3 Thierry Carval : CNDC (conductivity) valid min is set to 8.5 instead of 60.0 , ?2.2.3 Thierry Carval : vertical sampling scheme to manage profiles performed on different vertical axes , ?2.4 Esmee Vanwijk : meta-data format version 2.4 , ?2.2.3 Thierry Carval : global attributes and parameter attributes for CF compatibility stAll comments are welcome, before March 1. 14/02/2012 02:50 Comments from Annie Wong My main comments with this draft concern the multi-axis format for the single-cycle profiles files. I have attached an edited document here that contains the details. In addition, I'd like to draw your attention to the following: - This is profile format version 2.3, not 2.4. OK for 2.3 (a profile format version 2.3 was already described in the previous manual, however, no such file ever reached the GDACs). - The variable "vertical sampling scheme" is not optional. It is mandatory. This is the only means users have for finding out which profile is what. OK, changed 1 Argo data management n?12 and Argo trajectory workshop held in Seoul in November 2011 - Should "vertical sampling scheme" be STRING64 or STRING256? OK for STRING256 - I suggest you move the material on P.18 under "Notes on vertical sampling scheme" to P.13, and consider re-writing that introduction part. The rest can go into the netcdf variable specifications. Please see my doc. OK, I inserted your suggestions - PRES_DOXY in Reference Table 3 should now be removed. OK, removed. I do not remember how it appeared here. - In Reference Table 16, please assign specific descriptive names. "Bio-geochemical sampling" is not good. There are already oxygen, nitrate, chlorophyll, etc, being telemetered back to the DACs. I made some edits to Table 16. Please see my doc. Please note that as far as I know, right now all near-surface samplings are non-pumped. OK, I took your corrections. A last point concerns Reference Table 11. Please add Test 20 "Questionable Argos position test". Again, please see my doc. OK, added. 14/02/2012 05:15 Comments from Mizuho Hoshimoto Thank you very much for the new version. I have some comments. -page 25 Should "comment" of "DATA_TYPE", "FORMAT_VERSION", "HANDBOOK_VERSION", "REFERENCE_DATE_TIME", "DATE_CREATION", "PROJECT_NAME", and "PI_NAME" change to "long_name"? Yes, if I understand correctly the CF metadata convention, available at: On page 3, read the sentence: “Each variable in a netCDF file has an associated description which is provided by the attributes units, long_name, and standard_name. The units, and long_name attributes are defined in the NUG and the standard_name attribute is defined in this document.” See also page 11. -page 27 Don't you add "standard_name" and "axis" to "JULD", "LATITUDE", and "LONGITUDE"? Sorry, I added these missing attributes -page 28 s will need "standard_name". I also added a standard_name to and _ADJUSTED -page 35 We need "STRING1024" in the dimension section. Added -page 36 The sample of "FORMAT_VERSION" should be <<2.4>>. Changed -page 38 Should "comment" of "PI_NAME" change to "long_name"? Yes, changed on page 38 and all the pages where “comment” exists without “long_name” -page 48 Don't you change "comment" to "long_name" at "DATA_TYPE", "FORMAT_VERSION", "HANDBOOK_VERSION", and "DATE_CREATION"? Or, should we remain technical data format this time? I changed all the pages where “comment” exists without “long_name”. This applies also to the technical data format. So I updated it to format version to 2.4. -page 58 About "FREQUENCY_DOXY" at table 3, the format checker of USGDAC says valid max is 25000.f, C format is %7.1, and fortran format is F7.1. I agree with the result of USGDAC format checker. The resolution should be 0.1f in that case. Also, "Aandera" should be "Aanderaa" Ok, changed. -page 61 The table 5 needs "I: Iridium positioning". In that case, "(ARGOS)" should be removed from the title. OK, I updated table 5. -page 64 Please add to the table 8 "844 ARVOR, Seabird conductivity sensor" and "853 Solo2, Seabird conductivity sensor". Table 9 also needs "IRIDIUM Iridium positioning system". OK, added. -page 66 Could you add "COOA" to table 12 for the objective analysis? OK, added. -page 70 Please change "Measurements at start of transmission" to "Measurements at start of transmission (TST)" for confirmation. OK, done. 15/02/2012 02:24 Question from Mizuho Hoshimoto I have a question. What is the difference between "FIRMWARE_VERSION" of profile file and "FIRMWARE_REVISION" of metadata file? If these are same, we should use same name. I feel "FIRMWARE_VERSION" looks better. I agree with you and think that "version" is more appropriate than "revision". I rename FIRMWARE_REVISION and MANUAL_REVISION into FIRMWARE_VERSION and MANUAL_VERSION. 15/02/2012 22:34 Comments from Claudia Schmid Thierry, Megan and argo-dm, attached are 3 files. argo_dm_usermanual_20120210_draft_comments.txt has suggestions and questions for most sections of the user manual. argo_dm_usermanual_20120210_config_CS.doc is a revised version of the configuration section. argo_dm_usermanual_20120210_code_CS.doc is a revised version of the measurement code table. Both 'doc' files contain suggestions and questions. 1.1.1 argo_dm_usermanual_20120210_draft_comments.txt I agree with Annie's changes to the beginning of the profile section. p.5 : JULD_LOCATION: 2nd column has more JULD than JULD_LOCATION no axis= "T" needed? I corrected the JULD:xxx attributes that should be named JULD_LOCATION:xxx There should be one time axis only in the file. The time axis is JULD, not JULD_LOCATION. VERTICAL_SAMPLING_SCHEME: 2nd column char VERTICAL_SAMPLING_SCHEME (N_PROF, STRING64); VERTICAL_SAMPLING_SCHEME:long_name = "Vertical sampling scheme of the profile" ; VERTICAL_SAMPLING_SCHEME:conventions = "Argo CTD sampling, Pump near 3rd column This variable is mandatory. Use vertical sampling scheme to differentiate and identify profiles from a single-cycle with different vertical sampling schemes. See Table 16. I agree with Annie on removing the "Note on vertical sampling scheme" on page 16. Table 16: 1 Continuous sampling Sampling continuously with bin-averaging 2 Discrete sampling Sampling at discrete pressure levels 3 Near surface, pump on Sampling near the surface, CTD pump on 4 Non-pumped near surface Sampling near the surface, pump off or secondary sensor package without a pump 5 Secondary continuous sampling Sampling continuously with bin-averaging with secondary sensors 6 Secondary discrete sampling Sampling at discrete pressure levels with secondary sensors 7 Unknown Sampling scheme is not known Explanations/motivation: Removed CTD in 1 and 2, because many floats sample P, T, S & O2 together. For the same reason I removed "Bio‐geo‐chemical sampling" and replaced it with 5 and 6. TC: I added this proposal as an alternative in the manual. Your proposal is more general, so probably easier to manage (we avoid a series of sampling schemes related to various parameters such as nitrate, chlorophyll, carbon…). We need to take a decision. Now I'll go beyond the basics (some of this might belong in the cookbook, but we do not have it yet, and it may be interesting for the user to find some more detail in the user manual). If a float measures P, T, S & O2 all of them would have NPROF=1, but it could be that: a) O2 is sampled for every second P, T, S (I'm just making this up) b) O2 is sampled discretely while P, T, S is sampled continuously (or vice versa) (I'm just making this up) c) P, O2 is sampled separately from P, T, S and interpolated T, S are used to get the density needed for O2 (this is real) How should such cases be handled? Suggestions (a) can use NPROF=1 for all. (b) maybe use NPROF=2 for O2. (c) use NPROF=2 for O2. TC: case a, b, c are ok form me. A remark on c: use flag 8 (interpolated) for T and S on the O2 profile (nprof = 2) Table 3: suggestion to change a few column names to match the names used on pages 17-18 in column 2. Parameter long name -> long_name Parameter standard name -> standard_name Unit -> units Valid min -> valid_min Valid max -> valid_max resolution is not shown in the table TC: ok for the change on column names, done I think that the parameter attribute "resolution" is a problematic. The parameter attributes are general, valid for all argo data files. However, the resolution may vary from one sensor to the other, it may change between real-time data and delayed mode data. Therefore, I think that we should remove the "resolution" parameter attribute. We need to take a decision. comment: do we need it (on p.17-18 column 2 as well as in table 3 column 4)? Profiles are in situ by default. If we delete it, then the other columns in table 3 become more readable. Some of the information in comment can go in the column long_name (which currently has the same entry as the column name for the oxygen-related variables). TC: I am ok to remove the "comment" attribute on parameters. Does someone disagree? Examples: VOLTAGE_DOXY in colum 2 could be replace with "Voltage reported by oxygen sensor" MOLAR_DOXY currently is described as "Technical value reported by sensors such as Aanderaa Optode" This looks more like an Oxygen concentration to me (based on the unit). So long_name could be "Molar oxygen concentration reported by the oxygen sensor" TC: OK, I also added the standard_name : mole_concentration_of_dissolved_molecular_oxygen_in_sea_water PRES_DOXY long_name could be "Sea water pressure at the depth of oxygen sampling" TC: ok, I added pres_doxy on table 3 The example on page 18 has no PRES_ADJUSTED (, ...) variables. TC: I removed PRES from the example. p. 19 Since we are changing so much, maybe we should change CALIBRATION_DATE to SCIENTIFIC_CALIB_DATE????? TC: ok The current trajectory format version is 2.2 -> we will be at 2.3 with the new one. Agreed? TC : ok for 2.3 for trajectory and profile formats p. 26 int MEASUREMENT_CODE (N_MEASUREMENT); MEASUREMENT_CODE:long_name = "Number referring to an action or measurement by a float that has a defined measurement code"; MEASUREMENT_CODE:conventions = "Argo reference table 15"; MEASUREMENT_CODE:_FillValue = 99999; Codes for actions or measurements are given in Argo reference table 15. Example: The number 1 is assigned to measurements made at the start of descent to the drift pressure. They could be time, location, surface pressure, etc. JULD_*_STATUS Regarding Megan's note: if we do not use the "0" STATUS, then a lot of the US floats will have "9" for many of the times. JULD_FIRST_STABILIZATION Julian day (UTC) of the first stabilization after the start of descent to the drift pressure. Megan, do you agree? TC 24/02/2012: accepted JULD_DEEP_DESCENT_START Julian day (UTC) of the start of the deep descent to profile pressure at the end of the drift phase. TC: Megan, do you agree? TC 24/02/2012: accepted JULD_END_TRANSMISSION Rename it to JULD_TRANSMISSION_END to match the renamed JULD_TRANSMISSION_START Same with it's STATUS. TC: ok, done CONFIGURATION_MISSION_NUMBER Could not find section 2.5.4 found it in section 1.11.5 TC: I changed 2.5.4 into 2.4.5, is that correct? CYCLE_NUMBER_ACTUAL just a thought: should we call this CYCLE_NUMBER_SHORT Why? Because both CYCLE_NUMBER and this variable contain the actual cycle number. TC: Megan, do you agree? TC 24/02/2012: I think that "actual" is more neutral than "short" (the name "short" may imply that there a "long" cycle_number, which is not true). Cycle number of the float. This variable will help to relate cycle timing information and grounding status to the measurments collected during each cycle. TC : Megan, do you agree? HISTORY_PREVIOUS_VALUE why are columns 1 and 2 green? TC: I don't know, there is no good reason, I removed the green. The current meta format version is 2.2 -> we will be at 2.3 with the new one. Agreed? TC: ok, changed Should the clock drift be in the trajectory file? TC: Esmee removed it from trajectory file toword the technical file with the name TIME_ClockDrift_DeciSECONDSperDay PLATFORM_MAKER Name of the manufacturer. Example : Webb Research Corporation TC: OK STANDARD_FORMAT_ID Lots of brainstorming on this one. a) this is not clear to me. Maybe it will be associated with the revision date available from many manufacturers? If so, the combination of platform maker, type, transmission system and firmware revision, manua; revision will fully describe a float type. The corresponding documentation could have a standardized file name that consists of all the listed information. Maybe the STANDARD_FORMAT_ID could be replaced with FLOAT_MANUAL_FILE_NAME (without giving the extension). This file could then be made available through some web page (and it could be found with a search engine). b) if host site not yet determined, then what should we do when we start implementing the new format? TC: Esmee, can you comment? DAC_FORMAT_ID do we need this if we have a STANDARD_FORMAT_ID TC: Esmee, can you comment? Suggestion: present the PLATFORM_FAMILY, PLATFORM_TYPE, PLATFORM_MAKER, FIRMWARE_REVISION, MANUAL_REVISION right next to each other, prefereably in the given order. This could be followed by FLOAT_MANUAL_FILE_NAME or STANDARD_FORMAT_ID TC: done FLOAT_OWNER The owner of the float (may be different from the data centre and operating agency). Is this supposed to be filled with the name of a person or with the name of an agency? How does it relate to the PI name (which is what I associate with the float owner). Do we need a PI_INSTITUTION? TC: Esmee can you comment and give an example ? AGENCY versus INSTITUTION. In most (or all) cases we used INSTITUTION. So, maybe we should use OPERATING_INSTITUTION instead of OPERATING_AGENCY. What is an OPERATING_AGENCY (example)? TC : ok for institution instead of agency. Esmee, can you provide examples for each "Example:…" ARGO_GROUP what this means is not quite intuitive. Could not come up with a more meaningful name. Any ideas anybody? TC: … END_MISSION_STATUS Seems like if no dimension is given the length is 1 by default. True? TC: I think it's true. For 3. column: maybe add "T" -> No more transmission received, "R" -> Retrieved. I know it is in column 2, but some users may not see it there. TC: ok Should there be a distinction between retrieved at sea and retrival of beached floats? Should there be a distinction between grounded and all other cases of no more transmission received? TC: Esmee can you comment? CONFIG section see argo_dm_usermanual_20120210_config_CS.doc TC: Esmee can you comment? OK for me. p. 43 ARGO_GROUP (see above) BATTERY_TYPE - we'll have to permit "unknown" BATTERY_PACKS - we'll have to permit "unknown" DAC_FORMAT_ID - is string 16 long enough? (related to float characteristics section) MANUAL_REVISION - we'll have to permit "unknown" PTT - we'll have to permit n/a for Iridium and Orbcomm floats TRANS_FREQUENCY - we'll have to permit "unknown" TRANS_SYSTEM_ID - what is this for Iridium and Orbcomm? TC: "unknown" means "we did look for this information, we could not find it and we think that we won't be able to find it" Is that correct Esmee ? p. 57-58 do we need the sub-section "Remark on unit conversion of oxygen"? I think not. By removing it we avoid the potential for incosnsitencies with the more detailed oxygen documentation on the ADMT web page. TC: you are right; this remark has to be removed. The instructions to manage oxygen will be detailed in the cookbook. p.67 (table 15) suggestion for the first paragraph in this section: In the trajectory file, each action of the float during the cycle is associated with a code (measurement_code). One or more measurements might be taken at the time of the action. The code allows matching the measurements with specific times and actions during each cycle. TC: I added you suggestion. measurement code table: see argo_dm_usermanual_20120210_code_CS.doc TC: Megan, do you agree on the update of the " Measurement codes table" ? TC 24/02/2012: Megan agreed, table updated 16/02/2012 06:59 Comments from Kanako Sato Thank you very much for the new version. I have some comments. 1. page 38 STANDARD_FORMAT_ID and DAC_FORMAT_ID are not clear to me. What are we supposed to assign to these parameters? Esmee proposes 2 tables to list and describe the valid formats. Is that correct? Wouldn't you prefer 2 reference tables in the user's manual ? 2. page 44 Could you change 'float SENSOR_ACCURACY(N_PARAM)' to 'char SENSOR_ACCURACY(N_PARAM, STRING32)'? 'Processing Argo OXYGEN data at the DAC level version 1.2' says that SENSOR_ACCURACY for aanderaa oxygen sensor (Optode 3830/4330/4330F) must be assigned '8 micromol/L or 5%'. So, data type of this parameter should be not 'float' but 'char', and the dimension should be '(N_PARAM, STRING32)'. OK, done. I understand that we need to give a unit in addition to the accuracy value. I updated SENSOR_ACCURACY and SENSOR_RESOLUTION into character strings. 3. page 45 As Claudia said, we will have to permit n/a for Iridium floats. OK 4. page 64 Could you add 'Iridium' to table 9? At ADMT#12, it was agreed that a weighted average of all Iridium fixes should be included in the profile file if we cannot receive GPS position. Because our NEMO-IR floats cannot often get GPS postion, we would often assign 'Iridium' to POSITIONING_SYSTEM in the profile file. OK, added. 20/02/2012 Message "draft n?2 of the next Argo user's manual, Seoul update" from Thierry Carval Thank you very much to Annie, Mizuho, Claudia and Kanako who sent feedbacks and comments on the user’s manual Seoul update. I initiated a document (argo-dm-user-manual-seoul-update-comment.doc) containing these comments. Megan and Esmee, there are some questions for you (green characters). I started to update the manual according to these comments. Click here for the comments and the draft n?2. Click here for the comments and the draft n?2. 20/02/2012 23:10 comments from Annie Wong This email contains some minor comments for Users Manual draft #2. Because the comments are only minor, I am only sending them to you and not to argo-dm. Please incorporate them in your draft #3. Thank you. -page 14, please edit: "For example, all Argo floats collect at least one profile [per cycle] that contains the CTD measurements [delete]". TC: done -page 21, "SCIENTIFIC_CALIB_DATE", not "SCIENTIFIC_CALIBRATION_DATE". TC: done -page 58, www link to the QC Manual is now at: "www.argodatamgt.org/Documentation". TC: done -page 68, "hexadecimal", not "hexidecimal". TC: corrected -page 68, in the www link, "argodatamgt", not "argodtamgt". TC: corrected 21/02/2012 00:31 comments from Annie Wong I am concerned with Reference Table 16: vertical sampling scheme. This is the table that defines how to store multiple profiles from a single cycle. As you can see from Thierry's draft no 2 (page 75), an agreement is yet to be reached. I have attached here a proposal for Table 16 that incorporates suggestions from Claudia in her previous reply to argo-dm. I urge all Argo groups that have speciality floats to please contribute your thoughts. In thinking through this proposal, I consider the following list as essential qualities of this multiple-profile scheme. 1. The primary Argo CTD profile (and all measurements taken at the same levels) should always be profile number 1. 2. All other profiles will be recorded as profile number >1, but the number is not fixed. For example, some DACs will have "Near-surface sampling" as profile number 2; some DACs will have "Secondary sampling" as profile number 2. 3. The code for each sampling scheme needs to be an agreed and fixed character string, so that it can be searched by users. Because of this, I propose we keep it as STRING64, not STRING256. 4. The code for each sampling scheme should be general, so that all parameters collected under that scheme can be included. For example, "Primary CTD sampling" is misleading because, as pointed out by Claudia, there are DOXY that are sampled at the levels of the primary CTD profile. Likewise, let's leave out specifications such as "continuous" and "discrete". There are many Iridium profiles that are half continuous and half discrete. 5. Let's not list "Unknown". All sampling schemes should have a code and a description. 6. There is no "pumped" near-surface sampling. 22/02/2012 20:56 comments from Megan Scanderbeg Thank you for putting all of this together for me. The trajectory format section looks very nice. I have not received any other feedback on the trajectory file format draft I sent around, so I think it is complete for now. I only saw one question for me which was on page 72 asking about the wording right beneath the reference table 15. The wording is fine with me. All the other green changes in the trajectory related sections look good. Thanks, and let me know if I missed something you want me to review. TC: table 15 contains now Claudia's updates 23/02/2012 02:45 comments from John Gilson I think Annie's table moved things in the right direction, but perhaps a bit too far. I guess I fall somewhere in between Claudia and Annie. I agree with Annie's points 1(mostly),2, 3(mostly) and 5. Of the others.... 1) See 4b 3) I like the flexibility of having more characters, since the future is unknown. At present 64 seems like enough, but who knows in the future. TC: is 256 better? From Breck's comment, I think that we shall have to describe accurately the sampling scheme. 4) I believe that we cannot ignore the method of sampling in this variable. This is where this information is best listed. It is rather simple to include sub-categories in the first row for "Primary Sampling [Continuous]", "Primary Sampling [Discrete]", and "Primary Sampling [Mixed]" (see attached doc). It is just as easy to search. If all you care about is "Primary Sampling" you can ignore the latter portion, else it is information that should be available here. 4b) As a corollary to 4)...if we define the method of sampling in the 'Primary' data retrieval, another instrument (such as DOXY) which samples by a different method then the 'Primary' should be in the "Secondary Sampling" camp. For instance if DOXY is measured at the same pressure levels but it is a point measurement versus an average, it should be moved away from 'Primary Sampling' and into 'Secondary Sampling'. Annie's point 1) would hold with the added statement that "(and all measurements taken at the same levels and by the same method)". 6) I'm sure there is a current definition of 'Near-surface sampling' that everyone has in their minds. But I do not think we should limit it to un-pumped. Currently the SOLOII records a pumped spot sample of P,T,S at about 0.7 dbar. It should not go into the 'Primary sampling' profile (which is an averaged measurement). In the future the SOLOII might get measurements even closer to the surface. It may return a pumped 1Hz profile near the surface. My point is that we just don't know what the future holds. As an example of the flexibility necessary, the SOLOII firmware has just been modified to return a secondary discrete profile (primary is averaged), changeable by command of the operator through 2-way communication. The initial reason for this was a check on the primary data, but there certainly are scientific reasons to get this data back as well (for instance near surface). Its a new world out there with Iridium, and it would be easiest to make this table as flexible as possible while allowing for as easy of parsing as possible. I've attached a modification of Annie's table that I would envision as a good solution. TC: I am fine with your proposal, Annie and Claudia also agree. I replaced table 16 with your table. A remark: I am not comfortable with the column name "Profile #" . I think that this is not a "profile number", it is a "Sampling scheme number". 23/02/2012 04:19 comments from Annie Wong I like your Table 16. Thanks. I also agree with your points. The only question I have now is with regards to "Bounced profiles". I am not familiar with these and can't comment on whether they fit into "Secondary sampling", or whether they need a 4th entry of their own. Can someone who is familiar with these "Bounced profiles" provide some thoughts, or volunteer a code and a description if needed? 23/02/2012 15:45 comments from Claudia Schmid Annie and John, We are getting there... Some information on bounce floats and a question/suggestion for Table 16 (John's version). 1) bounce profiles. These are usually 7 profiles that all follow the same sampling scheme (not sure which on, but likely to be discrete). "Secondary sampling" may work just fine for them. Regarding "Profile #": Since the bounce profiles have their own cycle number (i.e. the floats do a normal profile in cycle N and the bouncing profiles in cycle N+1), the profile number within a bouncing profile file will be 1-7. The >1 in the "Profile #" column for "Secondary profiles" does not allow this case. 2) do we need [unknown] under "Code"? Hopefully not, but for old floats this may well be the case. Also, DACs will most likely need time to find out/verify what each float type does. 24/02/2012 05:51 comment from Breck Owens Annie - I hate to muddy the waters, but I think it is important that we find a way to document which samples are "discrete" and which are "continuous," which I take to be bin-averaged. TC 24/02/2012 12:02 comment from Thierry Carval Hi All 4, I understand that Breck would like a way to record the details of the sampling scheme. These details may be added as described in the attached document. The vertical sampling scheme variable is no more a simple code, but a code with details enclosed in brackets ([]). The STRING256 may not be enough. Is that realistic? I prefer to ask your opinion before sending this to the whole mailing list. I would also like to have 2 more examples (Solo and Apex in addition to Probio). 25/02/2012 00:31 comment from Annie Wong TC: see answer from John Thierry, that is an impressive effort. But I think it's unrealistic. Breck's wish to see a distinction between bin-averaged data and discrete data is for all Argo floats. Therefore the best place to do this is to invent a new metadata variable (e.g. "vertical_sampling_method") in the meta-data files. This "vertical sampling scheme" variable that is discussed here is not the right place for this information, because this variable is only available to a small number of floats (namely those that produce multiple profiles in a single-cycle) - it is not available to all Argo floats. 25/02/2012 01:20 comment from John Gilson I believe that what Thierry presented is very close to what was discussed and had agreed upon at the ADMT. It is my hope that something similar to it is what is finally adopted. I hope to have a proposal document to share with the group next week. Regarding Annie's comments: For there to be a metadata variable of "vertical_sampling_method" in the meta data file, just as useful in an Iridium world as the proposed variable in the profile file, it would likely need to be dimensioned as (N_CYCLE, N_PROF). Since the Meta file data does not contain these dimensions, it would be a major change to that file. This information could also possibly be added to the CONFIG variables in the metafile however the CONFIG variables are mainly for trajectory information and the decoder variable (CONFIGURATION_MISSION_NUMBER) resides in the tech file, not the profile file. Or it can go in the profile file, where it would lie right next to the data and be easily accessible. What I proposed is an addition of a single word to the sampling scheme names. The addition of this word does not seem troubling to me. Perhaps I'm overlooking something. I like Thierry's [description] idea, but I guess I would lean towards it being optional. My hope would be that reported profile data being averaged or discrete would be available for all cycles as it is important information. I might add, I don't recall the idea that this variable cannot be in a profile file if N_PROF=1. It was discussed that we wouldn't force DACs/PIs to reprocess N_PROF=1 files just to add the variable. However I don't recall that it was stated that it couldn't be in N_PROF=1 files. I feel that it is important enough for SIO SOLO that I would add the variable to all my files. I don't believe that many people understand that the SIO SOLO reports binned Temperature and Salinity. My understanding, after the ADMT, that the "vertical sampling scheme" variable would be added to new floats data files....and that if at some point all files are rewritten (for some reason), it would likely be applied backwards (if possible). 25/02/2012 20:01 comment from Breck Owens Annie - I agree with your suggestions. Breck 27/02/2012 16:00 comment from Claudia Schmid Hi all, seems to me that a solution could be to have general categories in the meta file (continuous, discrete, mixed, varying from cycle to cycle, or ...). For 'mixed' and 'varying', the more detailed information would be easiest accessible to the users if it is stored in the profile file. I'm not sure if I dare to suggest it, but the complicated vertical sampling schemes (e.g. 'mixed') may need a variable of the dimension N_PROF, N_LEVELS in the profile file. The problems with that are: (1) how do the DACs get this information? (2) what will be done if this information is not available? Then again, a descriptive way (see Thierry's example for a PROVOR) would pose essentially the same problems for the DACs. 28/02/2012 00:48 comment from John Gilson TC: ok with you John. I updated the chapter 3.16 "Reference table 16: vertical sampling schemes" with your table and examples. Following the past comments and posts regarding the inclusion of the 'Vertical sampling scheme' variable and table 16, below is my general hope on what this variable will represent. I've also included some SOLO II examples towards the bottom, similar to Thierry's example, but using this table. I know there are disagreements here, so I've again sent this to the smaller group hoping we can come up with some sort of compromise. First a general comment, I believe this variable should include information on the sampling scheme of the profile. Argo has added meta-like-information to the trajectory file (the CONFIGURATION_MISSION_NUMBER) which referenced variables in the METAFILE. Something similar could be done to the profile file. However, I don't think a description of the profiles data requires as much complexity. Thus the profile sampling scheme is easily added here, and of the most use to the user without opening multiple netcdf files. Thierry's initial Reference table 16 held rather specific names, while Annie's suggested table went very generic. For this table to be useful and parsable I would try something in between. I would argue that if there isn't detailed sampling scheme information in this variable, there is little reason to add the VERTICAL_SAMPLING_SCHEME variable to the profile file. If we think important only limited info, the CTD data could be forced into N_PROF=1 and then tell the user that any additional profiles can be identified via the STATION_PARAMETERS(N_PROF,N_PARAM) variable with new codes in the parameter code table (for example UNPUMPED TEMP). A user looking for non-primary CTD data would have to do a 2-dimensional search through an N_PROF,N_PARAM array, not a large change from what they must do now. I don't think that is the way to go however... So attached is my proposal for the variable Vertical Sampling Scheme through showing table 16, it is a modification of what I sent previous. I've made the table rather wordy, in order to point out subtle ramifications. I've tried to keep the mandatory parts of the Codes easily parsable, tried to limit the N_PROF to the fewest possible, and allow inclusion of as much information that the DAC might deem important. A few notes: a: With the exception of 'Near-surface sampling', I've kept the codes as general as possible. I'm assuming that the community finds this important enough to break into its own category. b: The name of the code and the nominal measurement type (averaged, discrete, mixed, ?) is mandatory. A full description is encouraged but optional. c: I changed 'continuous' as a measurement type to 'averaged'. While I would think there is a general correspondence between CP CTD and averaged bins, perhaps this is not the case. So I changed the name. d: I would differentiate between averaged and discrete measurement types by the number of polls of a sensor are performed to get the value transmitted. I would draw the line at >1 would be averaged and 1 would be discrete. However perhaps this is too strict. e: The Primary sampling profile must be N_PROF=1. No profile listed as N_PROF=1 can be labeled as other than Primary sampling profile. f: To limit N_PROF, it seems wise not to force people to create a N_NPROF>1 if the Primary sampling extends into the range of 'Near-surface sampling'. g: I've pulled the pressure range of 'Near-surface sampling' out of the air, please fill in something more appropriate. h: I added a 4th row that applies to the AOML 'bounce' cycles. What differentiates those floats is that multiple profiles are collected at different times, and assigned to a single cycle. So the profiles are different then 'Secondary sampling ', or the others which come from a single rise/fall. While iced-over floats might 'bounce' under the ice, I believe they assign the profiles to different cycle numbers. Also they complete a full drift in between profiles. And here are a couple of examples of how the SOLOII might look in the scheme I'm sending.... The SOLOII does not currently have secondary sensors. Thus what differentiates the N_PROF profiles on a SOLOII is the sampling method. Example for a SOLOII V1.2 float N_PROF=1: Primary sampling: averaged [nominal 2 dbar binned data sampled at 0.5 Hz from a SBE41CP] N_PROF=2: Near-surface sampling: discrete [Shallowest polling of a SBE41CP] Note: I'm aware it is likely unwanted to double the netcdf profile size to add a single point measurement. But it is an example. In near-future versions of the SOLOII, a full resolution profile over a set pressure range could be returned by the float. Example for a SOLOII V1.? float N_PROF=1: Primary sampling: averaged [nominal 2 dbar binned CTD data sampled at 0.5 Hz from a SBE41CP] N_PROF=2: Secondary sampling: discrete [1 Hz sampled CTD data from a SBE41CP] N_PROF=3: Near-surface sampling: discrete [Shallowest CTD polling of a SBE41CP] Note: According to table 16, I could have placed both N_PROF=2 and N_PROF=3 under Near-surface sampling discrete, because the sampling scheme is the same, even if the N_PROF=2 profile is deeper (for example targetting the base of the mixed layer during a hurricane). I split them here as an example of N_PROF > 2. 29/02/2012 00:35 comment from Ann Thresher Hi, Thierry, I like this plan - it's very clear and covers everything, if we decide to put this in the profile files. Annie's objection that this would only be available for multi-profile cycles would be solved by rewriting all profile files to include this variable - not a major headache but a bit of work would be required and, like John, I would do this even if it were not 'required'. It would be nice for all files to be more or less identical in structure. I do agree it will need to be dimensioned by the number of profiles in the file, however - so that each profile in a multi-profile file has a specific descriptor. And I agree - string 256 might not be large enough to hold all the possible variations. If we get some more input from people who work with the other float types, that might help us decide. I will take a look at our Apex floats but it won't be until later this week so hopefully you will get some input from someone else before then ;) I have wondered if this needed to be in the mission description section of the metadata files because that is what this is and we could easily cover it with additional variables. However, having it intimately connected to the profile also makes sense and would be easier to use in some ways. Metadata missions do not need to be restricted to trajectory variables (John's objection) - these are simply the variables we have identified so far. Adding other variables is simple and makes it easier to 'reuse' schemes. In many cases, a single set of sampling-scheme variables would be required associated with a single mission number. I suggest that the vertical sampling scheme would not change very often even for multi-mission floats. And we can make a similar distinction between primary, secondary and near-surface variables easily in the metadata mission configuration section. Also, I have always thought that the profile files also needed the mission number added to them but that WOULD require rewriting all of the profile files. Something to consider adding if we decide to rewrite these files...? One correction - the decoder variable (CONFIGURATION_MISSION_NUMBER - probably should be CONFIG_MISSION_NUMBER for consistency, however) resides in the traj file as a new variable where it is needed, not the tech file which will not contain configuration variables. TC: Yes, I renamed all the CONFIGURATION_MISSION_NUMBER into CONFIG_MISSION_NUMBER. BUT - I would NOT want to see this in the metadata files as suggested by Annie, if it required continual rewriting of those files. We really don't want do this when we have other files that can more easily provide this information. I guess in the end I agree with John - it's important and, though we would need to rewrite our profile files, we've done this before and it's not that hard. So, assuming we're putting this in the profile file and not as part of the mission parameters, Thierry's scheme works for all. TC: ok, that is the proposal described in the 14/03/2012 manual. 29/02/2012 03:29 comment from Annie Wong TC: ok, corresponding proposal sent on 14/03/2012 Hi all, Antoine Poteau visited UW for a couple of days. His group is putting out floats that carry additional sensors to measure fluorescence, chlorophyll, etc, with fairly complicated sampling schemes. We looked at Thierry's example, and noted that the "description" part of the character string will become very long for some of the speciality floats. For example, in Thierry's chlorophyll example there are 3 vertical divisions. In future there could be 5 or more vertical divisions. We feel that the meta data file, under the multiple configuration scheme (CONFIG_MISSION_NUMBER?), is a more natural place to store this information. Lastly, we feel that Thierry ought to send this example to argo-dm, so that the specialists and DAC managers can comment on it. 29/02/2012 08:52 comment from Thierry Carval Hi Annie, I agree with you, the byzantine details of the sampling schemes and measurement methods should be placed in the metadata file. So I try (by the end of the day) to send an update of the user's manual with John's proposal for the profiles vertical_sampling_scheme. In addition, I will mention the issue to record the details of the sampling schemes in the metadata section. TC: the proposal sent on 14/03/2012 will focus on the description of the vertical sampling scheme in the profile file, not in the metadata file. 01/03/2012 05:09 comment from Annie Wong Hi All, I have some comments on the changes to the draft manual for the metadata section that Thierry circulated and the follow-up suggestions. 1. The new metadata format version should be 2.4 as the previous draft manual (June 14th 2011) refers to the two previous versions of the metadata format (2.2 and 2.3). TC: ok 2. I'm happy to change the variable names FIRMWARE_REVISION to FIRMWARE_VERSION and similarly MANUAL_REVISION to MANUAL_VERSION as per Mizuho's suggestion. TC: ok 3. Claudia circulated a rewording of the metadata section (for configuration parameters). I'd like the draft to remain as it is in my version and Thierry's (but am happy for the edits that Thierry has already accepted to be included). Claudia - my main concern is that your re-wording changes the implementation of the scheme that was agreed at ADMT. The idea of having mission configurations is that similar sampling schemes can have the same mission number and be easily identified, i.e. separating shallow from alternating deep profiles. We specifically don't want to recommend in the manual that we use a new mission number for each cycle as this defeats the intention of the scheme. (I realise that you will probably do this anyway for some of your floats - but we need to keep in mind this is not the way the scheme is meant to be implemented and we should not be recommending this to users in the manual). TC: ok Secondly, to clarify the CONFIG_MISSION_NUMBER variable - it is immaterial whether the data is transmitted by the float or not. The mission 0 parameters are intended to be those that do not change for the life of the float.Mission 1 to N parameters are those that are changeable (they don't have to change but the point is that they are potentially changeable) during a float's lifetime. So for a float with a basic mission that does not change the way it samples it would have both mission 0 (the pre-deployment or launch variables) and mission 1 variables. In the trajectory files, no float will have a CONFIG_MISSION_NUMBER = 0. Mission 0 information will only be contained in the configuration section of the metafile. Floats with a single(basic) mission will report only CONFIG_MISSION_NUMBER = 1 in the traj file. Floats with multiple missions will report CONFIG_MISSION_NUMBER = 1,2,3,...N. 05/03/2012 11:59 comment from Breck Owens Answer to Thierry's message "On Feb 29, 2012, at 10:52 AM, Thierry Carval wrote:" I too agree that this is the right place to describe the sampling schemes. 05/03/2012 17:31 comments from Claudia Schmid TC: see "14/03/2012 00:59 comment from Esmee Vanwijk" Esmee and All, > 3. Claudia circulated a rewording of the metadata section (for configuration parameters). I'd like the draft to remain as it is in my version and Thierry's (but am happy for the edits that Thierry has already accepted to be included). Claudia - my main concern is that your re-wording changes the implementation of the scheme that was agreed at ADMT. The idea of having mission configurations is that similar sampling schemes can have the same mission number and be easily identified, i.e. separating shallow from alternating deep profiles. We specifically don't want to recommend in the manual that we use a new mission number for each cycle as this defeats the intention of the scheme. (I realise that you will probably do this anyway for some of your floats - but we need to keep in mind this is not the way the scheme is meant to be implemented and we should not be recommending this to users in the manual). I'm not exactly sure what is meant by this. 1) I was trying to make things more precise and more easily understandable by users who are not in the middle of things. 2) Since it can happen that the mission number could be changing with every cycle, I think that users should be made aware of that. > Secondly, to clarify the CONFIG_MISSION_NUMBER variable - it is immaterial whether the data is transmitted by the float or not. The mission 0 parameters are intended to be those that do not change for the life of the float.Mission 1 to N parameters are those that are changeable (they don't have to change but the point is that they are potentially changeable) during a float's lifetime. So for a float with a basic mission that does not change the way it samples it would have both mission 0 (the pre-deployment or launch variables) and mission 1 variables. In the trajectory files, no float will have a CONFIG_MISSION_NUMBER = 0. Mission 0 information will only be contained in the configuration section of the metafile. Floats with a single(basic) mission will report only CONFIG_MISSION_NUMBER = 1 in the traj file. Floats with multiple missions will report CONFIG_MISSION_NUMBER = 1,2,3,...N. 3) regarding the definition of mission number 0: I looked at the ppt 关于艾滋病ppt课件精益管理ppt下载地图下载ppt可编辑假如ppt教学课件下载triz基础知识ppt x file of the new configuration developed at the ADMT meeting. It showed a mission number -1 as the initial (pre-deployment) configuration and then started counting mission configurations as 0, 1, 2, ... For the ADMT meeting report the column with -1 was removed and the mission numbers start with 0. However, there was no explanation as to what the mission 0 represents. Both schemes (with or without the -1) would allow storage of the initial configuration. This background plus trying to make life easy for users is what motivated my suggestions. 05/03/2012 22:07 comment from Annie Wong TC: see "14/03/2012 00:59 comment from Esmee Vanwijk" Esmee, I think it is essential that Claudia's re-wording for the multiple configuration scheme be included in the users manual. I am referring to the part that "If a float [with two-way communication capability] has many configuration parameters that are transmitted with every cycle, then a new mission number can be used for each cycle." There are, and will be, complex cases where this implementation makes the most sense. It doesn't defeat the intention of the scheme, which, at the core, is simply a data format that allows multiple recordings of certain parameters. 06/03/2012 20:22 comment from John Gilson TC: see "14/03/2012 00:59 comment from Esmee Vanwijk" I agree with the idea of Claudia's re-phrasing as well. I believe she is pointing out that every cycle might have a new mission if a PI makes continuous changes. I might extend that idea further. In the case of a DAC unable to report changes in mission for a float...or simply doesn't, I'd prefer Argo guide the DAC to default to a new mission number every cycle, over the alternative of a constant mission number. I don't believe this variable is intended to remove the users responsibility in examining what the best definition of a mission for their particular scientific question. In the manual I would state that Argo suggests a minimum of configuration missions, but the Argo program leaves it to the PI to determine what best constitutes a mission in setting this variable, thus a float may range from having a single configuration mission for its entire lifetime to a new mission every cycle. 09/03/2012 03:27 comments from Ann Thresher TC: see "14/03/2012 00:59 comment from Esmee Vanwijk" HI, John and Annie, Esmee and I are working on some changes to the user manual but there are several issues to be addressed. As suggested, we have clarified in the manual that, if the mission changes every cycle, then, of course, a new mission number would be required for each profile (this is assuming that all missions are unique). Annie stated "If a float [with two-way communication capability] has many configuration parameters that are transmitted with every cycle, then a new mission number can be used for each cycle." - we have real problems with this when the parameters aren't changing. However, if the parameters do change for each mission then she is right. The tricky thing here is whether these changes repeat earlier settings. Let's face it - how many PIs change the mission totally EVERY cycle. We suggest rewording it thus: "If a float [with two-way communication capability] has many configuration parameters that change with every cycle, then a new mission number should be used for each cycle." Right from the start, I've had a hard time believing that detecting a mission 'change' is that difficult - you have the metadata file, you have the mission parameters, a simple parsing of values would tell you what, if anything, has changed. But you can't force people to play by the rules. We just want to make sure that it's clear what the rules/intent are and not allow too much weasel room. Therefore, adding a sentence giving blanket permission to just lazily use a new mission for each profile would be dumb. This insistence that somehow a transmitted mission parameter is different from one that is not transmitted is mystifying. Parameters can change and not be transmitted, they can remain the same and be transmitted every cycle. The treatment we agreed upon was very specific. If something changes and has not been used before, it's a new mission. If it changes and mirrors a previous mission, then we re-use that mission number. If it doesn't change, then whether or not it is transmitted is IRRELEVANT - it is the same mission as the previous profile. If someone wants to violate that process, using it in a way that is not consistent with the way everyone else uses it, then we can't stop them but we should be doing everything we can to make sure this is used as intended. If we aren't going to do this properly, then there's not much point in mission numbers. The whole idea was to help trajectory people (and others). With a little work, we can get this right and help everyone. The updated version of the manual pages will be sent later today or early next week (Monday is a holiday). 12/03/2012 12:28 comments from Mathieu Belbeoch TC 14/03/2012 : see Esmee answer "13/03/2012 05:47 comment from Esmee Vanwijk" Dear Claudia, Esmee and ADMT colleagues, Regarding your suggestions for new meta files variables: 1) Telecommunications. I reiterate that we do not need (and users even less) the Argos program (or contract) reference number. See TRANS_SYSTEM_ID I would also remove PTT and use two variables as follow: TRANS_SYSTEM TRANS_SYSTEM_ID and use multiple if it is the case (on gliders there are already the two telecom system, with Argos in backup. It will be certainly installed on expensive bio-floats). Examples Argos float TRANS_SYSTEM = ARGOS TRANS_SYSTEM_ID = 22507 Iridium float TRANS_SYSTEM = IRIDIUM TRANS_SYSTEM_ID = 123456 (public IMEI) Combo float TRANS_SYSTEM = ARGOS, IRIDUM TRANS_SYSTEM_ID = 22507, 123456 Finally, either we use a "DOWNLINK_AVAILABLE" tag somewhere or we use ARGOS, ARGOS3, ARGOS4, IRIDIUM_NEXT .. in TRANS_SYSTEM values. Questions: -some floats have Bluetooth connectivity (PROVOR, ARVOR). I think this works also with an id? Any more information on this? This connection is used for float set up and check up. Do we need to store this? -Do we need to know the air time provider ? IRIDIUM telecommunications can be served by ORANGE or CLS or any local provider. Do we use a TRANS_AGENCY ? -Where do we take care of RAFOS ??? Is it a new sensor ? 2) Manufacturers, Institutions, Agencies I would use one single reference table "AGENCY" to fill up PLATFORM_MAKER SENSOR_MAKER OPERATING_INSTITUTION (or OPERATOR) and any others I don't think we need both FLOAT_OWNER and OPERATING_INSTITUTION. References to individuals are inappropriate on the long run and we already have a PI Name. Maybe we could use the following variables: PLATFORM_AGENCY SENSOR_AGENCY OPERATING_AGENCY TRANS_SYSTEM_AGENCY To strongly show the reference to one single table AGENCY 3) STANDARD_FORMAT_ID, DAC_FORMAT_ID ... -What wording do we use: DATA_FORMAT_ID, or STANDARD_FORMAT_ID,... No preference here. -I think we should use a DAC_FORMAT_ID for a while, to help cross referencing, waiting to have all DACs aligned to a standard. 8 digits will be fine. 4)PLATFORM_FAMILY,TYPE, MODEL Good point to put all this together (see the on-line table for a start). We have two choices: - we keep PLATFORM_MODEL as it (free field) and we create two other standards: PLATFORM_FAMILY & PLATFORM_TYPE Or - we standardize PLATFORM_MODEL and add just one more PLATFORM_TYPE I prefer this one. 5)CONTROLLER_BOARD I think we should use here the multidimensional property of netCDF. CONTROLLER_BOARD_TYPE CONTROLLER_BOARD_TYPE_SERIAL_NO If FIRMWARE_REVISION is a date then it should have such data type. Maybe it should be related to the controller board. CONTROLLER_BOARD_TYPE CONTROLLER_BOARD_TYPE_SERIAL_NO CONTROLLER_BOARD_FIRMWARE_REVISION_DATE 6) PROJECT NAME We need here something standard between AIC/JCOMMOPS and meta files. All program names(chosen generally with Program Manager/PIs) are available on AIC website. Please update your files accordingly or suggest a new program name if needed. 7) MISC We need PLATFORM_MANUFACTURING_DATE SENSOR_MANUFACTURING_DATE ARGO_GROUP: I would use it with a reference table Core Equivalent Bio Coastal Do we add a Polar ? Are there any project funded under a "high latitude Argo" label ? DEPLOYMENT_PLATFORM should be standard (Ship name). It could use the SeaDatanet standard: I am finally worried with the FLOAT_SERIAL_NO that is not standard at all. We need provide guidance for each model. I will add the "serial_no template" to the on-line table. This is it for now. 12/03/2012 13:31 comment from Virginie Thierry Here is my answer to the following question: > Questions: > -some floats have Bluetooth connectivity (PROVOR, ARVOR). I think this works also with an id? > Any more information on this? > This connection is used for float set up and check up. Do we need to store this? From my point of view, there is no need to store this because it is used to program the float before deployment and once deployed, those information are useless. TC: ok 13/03/2012 05:03 comment from Esmee Vanwijk TC: see "14/03/2012 00:59 comment from Esmee Vanwijk" I'm about to circulate a new draft of the metadata section of the User's Manual but wanted to highlight the text relating to floats with multiple configurations first. In the interests of bringing this discussion to a close and catering to differing viewpoints I suggest the following paragraphs (see below) and hope that we can now finalise the changes to the manual. I'll circulate the extended version of that section shortly but the part I think most will be interested in is below. I hope these words accommodate everybody's concerns. It allows for the case of complex floats whilst reiterating that a minimum number of configurations is the preferred option. Regards, Esmee "If a float [with two-way communication capability] has many configuration parameters that change with every cycle, then a new mission number should be used for each cycle. Argo best practice and our recommendation to users, is a minimum of configuration missions; i.e. if there is a change to configuration parameters that does not repeat a previous configuration then a new mission number should be used. If the configuration parameters change, but mirror a previous mission then that mission number should be re-used. In extremely complex cases where mission changes are unclear, then a new mission number can be used for each cycle. Users should be aware that the metafile will need to be rewritten each time a new mission number is added." 13/03/2012 05:47 comment from Esmee Vanwijk Hi Mathieu and all, I've put in individual comments under each of your suggestions. >-----Original Message----- >From: Belbeoch Mathieu (JCOMMOPS) [mailto:belbeoch@jcommops.org] >Sent: Monday, 12 March 2012 10:28 PM >To: argo-dm@jcommops.org >Subject: RE: [argo-dm] draft of the next Argo user's manual, Seoul >update >Dear Claudia, Esmee and ADMT colleagues, Regarding your suggestions for >new meta files variables: >1) Telecommunications. >I reiterate that we do not need (and users even less) the Argos program (or contract) reference number. >See TRANS_SYSTEM_ID I suggest we keep TRANS_SYSTEM_ID, it's currently being populated and costs us nothing to keep. If we think ahead to potentially retrieving all the data for a particular program from CLS (or others) in 20 years time then keeping the contract number might be useful. TC : OK >I would also remove PTT and use two variables as follow: >TRANS_SYSTEM >TRANS_SYSTEM_ID >and use multiple if it is the case (on gliders there are already the two telecom system, with Argos in backup. It will be certainly installed on >expensive bio-floats). Currently PTT and TRANS_SYSTEM_ID are two different variables so I'm not sure why we'd get rid of PTT/or confuse the two? PTT is the transmission identifier of the float while TRANS_SYSYTEM_ID is the program identifier of the telecommunication subscription, e.g. each contract (or DAC/program etc) has its own identifying number. There's no reason why we can't dimension this if there's more than one transmission system. In this case we need to edit the manual to change the dimensions for TRANS_SYSTEM and TRANS_SYSTEM_ID. TC : ok >Finally, either we use a "DOWNLINK_AVAILABLE" tag somewhere or we use ARGOS, ARGOS3, ARGOS4, IRIDIUM_NEXT .. in TRANS_SYSTEM values. I think it's useful to distinguish between future evolutions of transmission system, e.g. ARGOS3 and ARGOS 4 etc. I don't think we need another variable, I suggest we allow the input to TRANS_SYSTEM to include things such as ARGOS3 and IRIDIUM Next. However we do need to have a standard reference table to get the naming conventions standardised and I also suggest that we make "ARGOS" and "IRIDIUM" the first part of each of these standard inputs so that we can still search for a generic ARGOS and pull out ARGOS, ARGOS3 and ARGOS4 etc. TC : ok >Questions: >-some floats have Bluetooth connectivity (PROVOR, ARVOR). I think this works also with an id? >Any more information on this? >This connection is used for float set up and check up. Do we need to store this? I agree with Virginie that we don't need to keep this information. TC : ok >-Do we need to know the air time provider ? >IRIDIUM telecommunications can be served by ORANGE or CLS or any local provider. >Do we use a TRANS_AGENCY ? I think we should keep the metafiles as simple as possible to start with, so I don't think we should add this. TC : ok >-Where do we take care of RAFOS ??? >Is it a new sensor ? RAFOS is technically a sensor/measurement package in terms of the RAFOS receivers that are attached to some under-ice floats but in this context it is a positioning system that provides a post-processed position for under-ice profiles. RAFOS equipped floats still have a primary positioning system (e.g. GPS) but when they are under ice and unable to surface, the RAFOS signals can be used to determine a position. I imagine that it would work in this way.. The POSITIONING_SYSTEM variable in the metafile will need to be dimensioned by the number of positioning systems, i.e. GPS, RAFOS. Then, in the profile file the POSITIONING_SYSTEM variable would be filled with whichever system was used to provide the position for that profile, i.e. for ice free profiles it would be GPS. For the initial under ice profiles with missing positions which are filled by interpolation the POSITIONING_SYSTEM would still be GPS but the POSITION_QC would be 8 to reflect that initial interpolated position. Once the RAFOS data is processed and a more accurate position determined, then the POSITIONING_SYSTEM would change to RAFOS and the POSITION_QC updated to reflect the accuracy of the RAFOS data. >2) Manufacturers, Institutions, Agencies >I would use one single reference table "AGENCY" to fill up >PLATFORM_MAKER SENSOR_MAKER OPERATING_INSTITUTION (or OPERATOR) and >any others I'm happy with having one reference table for both PLATFORM_MAKER and SENSOR_MAKER as they include manufacturers but OPERATING_INSTITUTION should be separate as it refers to the scientific institutes responsible for the float which are different from the manufacturers. TC: ok >I don't think we need both FLOAT_OWNER and OPERATING_INSTITUTION. >References to individuals are inappropriate on the long run and we already have a PI Name. This was discussed at ADMT12 and we agreed to add both of these variables, as some DACs made the point that the FLOAT_OWNER (note that this can be an institution, and may be different from the float PI) and OPERATING_INSTITUTION were different. TC: ok >Maybe we could use the following variables: >PLATFORM_AGENCY >SENSOR_AGENCY >OPERATING_AGENCY >TRANS_SYSTEM_AGENCY >To strongly show the reference to one single table AGENCY I would prefer that we stick with the names agreed at ADMT12 as everybody agreed on them. I don't think we need to add another variable for TRAN_SYSTEM_AGENCY. TC: ok >3) STANDARD_FORMAT_ID, DAC_FORMAT_ID ... >-What wording do we use: DATA_FORMAT_ID, or STANDARD_FORMAT_ID,... >No preference here. >-I think we should use a DAC_FORMAT_ID for a while, to help cross referencing, waiting to have all DACs aligned to a standard. >8 digits will be fine. These two variables are different and we need BOTH of them. DAC_FORMAT_ID is the ID that each DAC currently uses to identify the format of their floats. STANDARD_FORMAT_ID will be a standardised format number which enables cross-referencing between DACS. Both of these will be available in the online table. Mathieu - the STANDARD_FORMAT_ID is equivalent to your DATA_FORMAT_KEY in the table you sent around (can you please change the name on your online table so that it matches what we have in the files? - I'm happy to keep your 6 digit format). TC: ok >4)PLATFORM_FAMILY,TYPE, MODEL >Good point to put all this together (see the on-line table for a start). >We have two choices: >- we keep PLATFORM_MODEL as it (free field) and we create two other >standards: PLATFORM_FAMILY & PLATFORM_TYPE Or >- we standardize PLATFORM_MODEL and add just one more PLATFORM_TYPE I >prefer this one. This was discussed at ADMT12 and if I remember correctly, the consensus was that we add two new variables (which will have standard reference tables); PLATFORM_TYPE is the type of float, i.e. SOLO, APEX, PROVOR etc. PLATFORM_FAMILY is the category of instrument, i.e. Float, POPS, ITP. PLATFORM_MODEL is an existing variable that currently has non-standard input. Examples are: APEX_SBE, SOLO_FSI, PROVOR_MARTEC_FSI. It is my recollection that at ADMT12 we agreed that this information is available from other fields so we could delete this variable. If people wildly object to this, than we could leave it in as a free form field but strongly encourage users to populate this according to a standard formula. TC: ok >5)CONTROLLER_BOARD >I think we should use here the multidimensional property of netCDF. >CONTROLLER_BOARD_TYPE >CONTROLLER_BOARD_TYPE_SERIAL_NO This was also discussed at ADMT12 and my recollection is that various DACS preferred having separate variables (correct me if I'm wrong), i.e. CONTROLLOR_BOARD_TYPE_PRIMARY, CONTROLLOR_BOARD_TYPE_SECONDARY, CONTROLLOR_BOARD_SERIAL_NO_PRIMARY and CONTROLLOR_BOARD_SERIAL_NO_SECONDARY rather than using dimensions. TC: ok >If FIRMWARE_REVISION is a date then it should have such data type. >Maybe it should be related to the controller board. After feedback from various people we've agreed to have two new variables; FIRMWARE_VERSION (to make the name consistent) and MANUAL_VERSION. These are two separate variables; please see the manual for definitions. The FIRMWARE_VERSION is not always a date. We have float types where this is either a number or a string and therefore it should be string as data type. TC: ok >6) PROJECT NAME >We need here something standard between AIC/JCOMMOPS and meta files. >All program names(chosen generally with Program Manager/PIs) are available on AIC website. >Please update your files accordingly or suggest a new program name if needed. This is a highly desirable, not mandatory variable as not everyone will have a project name. No reason why we can't try and standardise this with a reference table. TC: there is no reference table for project name, and I do not think that we need it >7) MISC >We need >PLATFORM_MANUFACTURING_DATE >SENSOR_MANUFACTURING_DATE I think we need to keep the metafiles as simple as possible in the first instance, there are a lot of changes currently scheduled and we need to try and get the critical information populated first. I also recall a few people pointing out that they don't have this information and would not be able to populate the fields. Perhaps we could look at introducing these in the next round of metafile refinements if there's demand for it? TC: ok >ARGO_GROUP: I would use it with a reference table >Core >Equivalent >Bio >Coastal >Do we add a Polar ? Are there any project funded under a "high latitude Argo" label ? I agree we need a reference table. We could add Polar or High Latitudes, although it's worth noting that there are already a lot of high latitude floats out there that are funded under core Argo, so this could get confusing for users. I think the easiest way to separate out these floats is by position (i.e south or north of 60 degrees). TC: I added a reference table 17 "Argo group of floats" Polar floats are core Argo floats deployed in Northern or Southern locations, so I do not see a need for it. >DEPLOYMENT_PLATFORM should be standard (Ship name). It could use the SeaDatanet standard: > >I am finally worried with the FLOAT_SERIAL_NO that is not standard at all. >We need provide guidance for each model. >I will add the "serial_no template" to the on-line table. FLOAT_SERIAL_NO can't really be standardised as it is simply the serial number of the hull of the float and will differ depending on float type. However we can and do need to emphasise that this field should contain the number and the number only, i.e. not be prefaced with "SBE" or "FSI" etc. As background - this information used to be held in the variable INST_REFERENCE (now redundant) as the name was not clear and as it was populated with this type of mixed input i.e. SBE1679. Right, I think that covers everything! TC: ok 13/03/2012 09:27 comment from Kanako Sato TC: see answer "13/03/2012 21:26 comment from Annie Wong" Dear Annie, I'm so sorry that I send an email to you too late. (On Feb 21, 2012, at 2:30 AM) I think that it is important to store the method of sampling. So, I agree with John's points. Then, I have a question. What is the definition of "near-surface"? We JAMSTEC have several NEMO floats. They record pumped spot sample of P,T,S from 2000 dbar to depth shallower than 4dbar, for example, 2 and/or 0.5 dbars. I wonder if we should assign all P,T,S data from 2000 dbar to surface to "Primary sampling [discrete]" or not. For example, I assume that "near-surface" is the depth shallower than 4 dbar. For users searching "near-surface" data, it might be good to assign P,T,S data shallower than 4 dbar to "Near-surface sampling", even if the float record P,T,S from 2000 dbar to surface by the same sampling method. But if we attach importance the method of sampling, we might as well should assign all P,T,S data from 2000 dbar to surface to "Primary sampling". 13/03/2012 10:42 comment from Mathieu Belbeoch TC: see "14/03/2012 01:05 comments from Esmee Vanwijk" Hello Esmee, all, Thanks for your feedback. I am fine with your comments, except: 1) I think PLATFORM_MANUFACTURING_DATE, and SENSOR_MANUFACTURING_DATE are critical to better analyze float decay and problems. Today we generally use the deployment date for those analyses and it doesn't reflect manufacturing history. Sometimes platforms are deployed years after their manufacturing. We need to monitor this closely. I strongly recommend to start gather this information asap. 2) An attribute that is both a string or a date is useless with machine to machine processes. I would recommend to do two attributes: one date and one string. So they can be filled as appropriate considering different models. 13/03/2012 19:55 comment from Breck Owens TC: ok I have been on and off the net for the past few weeks. Will continue to have limited net access until I arrive in Paris on the 19th. I have quickly read John Gilson's suggestions and as usual I think he is pretty much on the money. 13/03/2012 21:26 comment from Annie Wong TC: John described near-surface sampling on chapter 3.16 "Reference table 16: vertical sampling schemes" Hello Kanako, Thank you for emailing in your comments - they are never too late. The original motivation for the single-cycle multi-profile format was simply to separate out measurements that are sampled on vertical levels that are different from the primary CTD. On the most practical level, this is to avoid the unfortunate situation of users associating exotic measurements with the wrong PRES. Recently, there are suggestions to expand this original motivation to include the need to separate out sampling methods. Almost all replies are agreeable to this suggestion. "Near-surface measurements" constitute a separate class because they satisfy the above criteria: different vertical levels and different sampling methods. So if you follow this definition strictly, then the entire NEMO profile from 2000 to 0.5 dbar will be "primary sampling". However, as a service to GHRSST, there are merits to include the NEMO shallow samples in the "near-surface" class, so users have all the near-surface data in one place. Justin can confirm that this is in fact how BODC arrange their ftp area for GHRSST users (ie. JMA shallow samples are included). I think this is a good idea, but I don't claim to speak for the majority. Does JMA have a preference? Justin - do you know what GHRSST would prefer? 14/03/2012 00:59 comment from Esmee Vanwijk Further to my email yesterday highlighting the new text for floats with multiple configuration parameters, please find attached the revised section of the metafile section of the draft User’s Manual. TC 14/03/2012: I reported your changes on chapter 2.4 "Metadata format version 2.3" 14/03/2012 01:05 comments from Esmee Vanwijk TC: this table is not yet include in the 14/03/2012 manual. Hi Mathieu and All, Thanks for sending round the data formats table. I have a few suggestions and questions: 1). the column PLATFORM_MODEL should be named PLATFORM_TYPE to be consistent with the way we’ve named the variables in the metafile and manual. 2). we also need to merge the table that I sent around earlier (attached again here) with your table. We could do this by adding in DAC_FORMAT_ID as a new column to your table and renaming your DATA_FORMAT_KEY to STANDARD_FORMAT_ID (this is the new standardised number that we are providing, I personally don’t mind if it’s a simple index (as in my version) or contains the PLATFORM_MODEL_KEY as in your version (do we need to have this as a separate column?). Perhaps all we need here is a reference key located in a handy place for users so that they know what platform number to assign to their floats. Bearing in mind also that this PLATFORM_MODEL_KEY is not currently a variable in the metafile (and probably doesn’t need to be). We should also cross-reference the decoders that are used for ANDRO. The other comment I’d make about data formats is that many different firmware revisions will have the same format type. This is why I decided to make my table version relatively simple (i.e. a basic cross-reference between DAC_FORMAT_ID and STANDARD_FORMAT_ID and leave the firmware version information to the individual DAC webpages, i.e: see the listings for JMA: You can see here that float type (equivalent to DAC_FORMAT_ID) = A1 has four different batches each with different revision dates but all of them have the same format. I’m still looking for feedback from other DACs as to whether this information is available online for their floats? The only web pages I’m currently aware of are: JMA: CSIRO: SIO: Could other DACS please let me know if this information is already online and if not, when we might be able to have access to it? Another consideration… is it possible to provide links to the electronic copies of the float manuals in your table? As I see it we have two approaches, we can either put all of this information into the one table as per Mathieu’s suggestion and live with multiple entries for each float type or we have a simpler table as per my suggestion which has just one example of each float type and then cross reference to an associated webpage for each DAC which then has the detailed listing of float types and serial numbers. The pros of Mathieu’s approach are that everything is in the one place however the disadvantage is that the table will become extremely long and we duplicate some information. Preferences or thoughts anyone..? Esmee 14/03/2012 05:38 comments from Mizuho Hoshimoto Hi, Annie, and all, I still think the difference between unpumped and pumped sampling near surface is important, so [pumped]/[unpumped] should be included after "Near-surface sampling" at the John's table. TC: pumped/unpumped can/should be added in the full description (between brackets). I added "pumped" in the Provor example on page 76, primary CTD sampling. In the optical secondary sampling, pumped or unpumped are irrelevant (no pump on the optical sensor). Also, it would be appropriate that the measurements with a same sampling method are added to "Primary sampling", even if there are some near-surface observations. Only the measurements sampled with a different method, for example, averaged and unpumped, will be added to "Near-surface sampling". I think many users will like to get measurements, which can be treated same way, at once. If GHRSST users need separate profiles, we might add same measurements to "Primary sampling" AND "Near-surface sampling". As Breck wrote, the meanings of "discrete" and "continuous" are not clear for me. 14/03/2012 10:04 comment from Annie Wong Hello Mizuho, I think Thierry is going to send out another draft of the users manual soon that has a revised Table 16 that John put together. I believe the revised version currently offers a choice of 3 single-word qualifiers: "discrete", "bin-averaged", or "mixed". Your suggestion of adding "pumped" or "unpumped" to near-surface sampling is good, and I hope Thierry can incorporate them into Table 16 if no one else objects. Description for the near-surface sampling class will need to be revised once JMA decides what to do with the shallow samples. 14/03/2012 Message "draft n?3 of the next Argo user's manual, Seoul update" from Thierry Carval Many thanks to the 10 contributors who sent 34 feedbacks and comments on the user’s manual Seoul update n?2. I updated the manual according to these comments. They mainly focus on chapter 2 "profile format" (vertical_sampling_scheme) and on chapter 2.4 "metadata format". I think that we have reached a general agreement on the manual's update. Click here to get the draft n?3 and its associated comments. Best regards, Thierry 14/03/2012 18:26 comment from John Gilson 15/03/2012 TC: I removed the first note and rephrased the second one. Hi Thierry, Thanks for all the work on the manual. My only comment is that you should probably remove my 'notes' in table 16. They are pretty wordy and not really appropriate for a manual. So perhaps remove...."Note: I'm aware it is likely unwanted to double the netcdf profile size to add a single point measurement. But it is an example. In near-future versions of the SOLOII, a full resolution profile over a set pressure range could be returned by the float. " And for the other note perhaps change it to... "According to table 16, both N_PROF=2 and N_PROF=3 could have been placed under the same profile number because the sampling scheme is the same." 15/03/2012 01:16 comment from Esmee Vanwijk 15/03/2012 TC: I inserted the changes on pages 44-45, page and added a new reference table 18. The "Configuration Parameter names.xslx" is listed on Annie - thanks for your comments. Thierry - I've made the changes Annie suggested and attached an updated doc based on your new draft 3. I've also attached an updated version of the Configuration parameter names table that needs to go up on the ADMT website under "Argo metadata files". Thanks, Esmee -----Original Message----- From: Annie Wong [mailto:awong@ocean.washington.edu] Sent: Thursday, 15 March 2012 7:33 AM To: Van Wijk, Esmee (CMAR, Hobart) Subject: Re: [argo-dm] Hi Esmee, I have 2 comments for the meta-data section. 1. At the end of the "multiple configurations" section, somewhere around page 43 or 44 in your doc, could you please add a brief paragraph that can tell people how to find out which cycle follows which mission? I understand this info is buried in the traj file. If you can put in a pointer in the meta section, that'd be very helpful. Otherwise, it'd be almost impossible for people to figure that out. 2. You should just go ahead and re-name Table 14b as Table 18, and ask Thierry to add a page for Table 18 in the users manual that tells people that these things are hosted on the admt www. annie 15/03/2012 17:50 comment from Annie and John TC 15/03/2012: I inserted the updated table. After discussing with John, I modified Table 16 to include Mizuho's suggestion for near-surface sampling, and replaced the hypothetical N_PROF=3 example with a real-life example that I think will better illustrate the intention of the scheme. My revised Table 16 is attached here. Please insert in the manual. 16/03/2012 03:52 comment from Mizuho Hoshimoto There are some small corrections. P.19 at "_ADJUSTED_ERROR" of profile format, "_ADJUSTED_ERROR:comment" should be removed. TC 29/03/2012 OK, done p. 36 at Dimensions and definitions of metadata format, comment, "String dimensions from 2 to 256" should be "String dimensions from 2 to 1024". There is same correction at dimentions and definitions of technical information format, "2 to 256" should be "2 to 128". It is too small correction! TC 29/03/2012 OK, done p. 60 at reference table 3, FREQUENCY_DOXY's C_Format should be "%7.1f" and FORTRAN_Format should be "F7.1". TC 29/03/2012 OK, done 29/03/2012 Message from Thierry Carval Many thanks to the 4 contributors who sent 4 feedbacks and comments on the user’s manual Seoul update n?3. The version 2.4 of Argo user's manual is now officially released. The new features, changes and updates of the manual were decided during the Argo data-management team meeting, held in Seoul in November 2011. Click here to get the version 2.4 of the user's manual (including the revision marks) and its associated comments. The official version, without revision marks is available here. Best regards, Thierry