Identification Quality Experiment Update
Thanks to everyone who signed up for the iNaturalist Identification Quality Experiment.
We're still getting the kinks of the study out before kicking this study into full gear, but we do have a small sample of 1,156 expert IDs to start playing with. Comparing the taxon suggested by the expert with the taxon associated with the observation just before the expert's suggestion, three things can happen:
1) match (88%): the expert's suggested taxon matches or is more precise than the taxon previously associated with the observation (e.g. observation is ID'd as Taricha or Taricha torosa and expert suggests Taricha torosa).
2) maverick (8%): the expert's suggested taxon is sister (maverick) to the taxon previously associated with the observation (e.g. observation is ID'd as Taricha torosa and expert suggests Ensatina eschscholtzii).
3) too precise (4%): the expert's suggestion is less precise than the taxon previously associated with the observation (e.g. observation is ID'd as Taricha torosa and expert suggests Taricha).
Two issues have became clear in these early days:
1) Ambiguous subjects: Because expert IDs are made blind, context like the description that observers often use to describe the subject of their observation may be missing. That can lead to situations like this where the expert thought the subject was the Laurel Sumac rather than the California Brittlebush. We're considering functionality to make the subject of an observation more explicit, but for now, skip observations if the subject is ambiguous:
2) Ambiguous disagreement: We are assuming that an expert's ID is the most precise ID that ANYONE can make based on the evidence provided. Thus, if an expert ID's an observation to genus that was previously identified to species, we're interpreting this as though the previous identification was 'wrong' by being too precise. Here's an example:
But since the expert IDs are made blind, the experts can't see how precisely observations are already ID'd. In some case, the expert's intent with a coarser ID was not to explicitly disagree with the more precise ID. For example, here's an observation was ID'd to the subspecies level and the expert added an ID at the species level. This species level ID was not intended to explicitly disagree with the subspecific ID, but thats how the system is currently counting it (ie the community's ID was 'too precise').
We're working on functionality to make it possible to distinguish explicit disagreements (e.g. 'no one can ID this to species from the evidence provided') from the alternative (e.g. 'I can only ID this to genus, but someone else might be able to ID it to species'). But for now, if you think someone else might be able to provide a more precise identification than you could, skip the observation.
Thanks again for your help. If you skip observations as described above (those with ambiguous subjects or when others might be able to make a more precise ID than yourself), then please proceed with making IDs for the experiment. I'll check back in when we have improvements to deal with these situations.
I'll also check in with more updates on the analysis as we have more data. Please contact me if you have any questions, concerns.
Thanks!
Scott
@charlie @fabiocianferoni @vermfly @jennformatics @d_kluza @arachnojoe @cesarpollo5 @ryanandrews @aztekium @lexgarcia1 @harison @juancarlosgarciamorales1 @garyallport @echinoblog @jrwatkin68 @bryanpwallace @wolfgang_wuster @bobdrewes