* Title: Full release
* Label: RELEASE
* Owner:
* Description:

Milestone for final release

* Requirements: *BETA, *TAXON-BACKPORT
----------------------------------------------------------------
* Title: Beta release
* Label: BETA

* Description:

Milestone for beta testing release

Not required, but desirable to have before beta launch:
  *TAXONSEARCH, *SEARCHCLEANUP, *SEARCHFIXES

"Requirements" and "desirable" lists provided by BP 200090208.

* See: BETA-prerequisites

* Requirements:
  *POP, *39M, *MONEX, *PASSWORD, *CHECKING,
  *TESTFLAG, *BUG
----------------------------------------------------------------

* Title: Unit tests
* Owner:
* Description:

Run the entire test suite.  Investigate all failures. Fix them.
When a test run too slowly, write a faster one.

When all tests pass, investigate test coverage.  Find all untested
methods.  Write tests for them.

----------------------------------------------------------------
* Title: Populate JRUAN schema with data from treebase 1
* Label: POP
* Owner: MJD
* Last-modified: 20090311

* Subtasks:
  A: Load matrix data
  B: Load tree data
  C: Build studies and analyses
  D: Add citation data
  E: Load taxon data 
  F: Load taxonvariant data
  G: Build taxonlabel table
  H: *MONEX
  I: *39M
  J: populate tree NTAX
  K: populate matrix NCHAR / NTAX

* Dependencies:
I -> A -> C
B -> C
C -> D
C -> G
E -> F -> G
I -> H
A -> K
B -> J

* Description:
Bill Piel supplied dumps of TB1 data, as follows:
1. Many nexus files representing input matrices
2. Many nexus files representing output trees
3. Three files describing taxa, taxonvariants, and taxonlabels,
   and mappings from Tl -> Tv -> Tx.
4. One file of citation data

All this data needs to be put into the JRUAN schema.
    util.BulkUpload is for A, B.
    util.AuxiliaryDataImporter is for C.
    util.CitationDataImporter is for D.
    util.LoadTaxonData is for performing tasks E, F, G.

* Current status: (last updated 20090223)
A - nearly all loaded
B - nearly all loaded
C - done except for those missing data from A/B
D - done except for studies missing from C
E - complete except for three Ribieria taxa
F - complete except as per E
G - complete except as per F
H - Rutger has code written somewhere, but not run
I - some resolved
J - done
K - done

* Other notes:
MJD should break this section into a series of sections, one for each
subtask.

Summary of loose ends as of 20090305 is in file POP-loose-ends.

C: Don't forget to clear the SUBMISSION.TEST flags when the
submissions are created!  All cleared 20090304

----------------------------------------------------------------
* Title: Missing original nexus files for matrices
* Owner: 
* Label: MONEX
* Last-modified: 20090209

* Description:
Because of a bug in the BulkLoader, the matrix data is in place, but
the NEXUSFILEOBJECT table has not been populated.  (This table records
the original nexus files from which each matrix or tree was created.)

Nexus files don't need to be parsed or understood.  They just need to
be stuck into the NEXUSFILEOBJECT.NEXUSCONTENT field whole.

Put them in.

Subtask A can start right away.  

Subtask B can't be complete until the matrices are all loaded (*POP.A)
but it can start before they are all loaded, and insert the nexus data
for the matrices that have been loaded.  Similarly, C can't be
complete until the trees are all loaded (*POP.B) but it can be run
incrementally until then.

Subtask D is independent of the others.

* Subtasks:
  A: Write special-purpose nexus file loader. 
  B: Run it on the matrix files. 
  C: Run it on the tree files.
  D: Is anything else missing?

* Dependencies:
  A -> B
  A -> C

----------------------------------------------------------------
* Title: What about the other 39 matrices?
* Label: 39M
* Owner: MJD
* Last-modified: 20090305

* Subtasks:
  A: Identify which 39 are missing
  B: Try to use BulkUpload to load each one.  Log failures.
     Save Mesquite logs!
  C: Analyze log files.  Fix obvious problems.  Repeat B-C as needed.
  D: Refer remaining files to Bill Piel.
  E: Implement Bill's advice.
  F: Try to reload missing tree files.
  G: Make sure all trees are actually in.
  H: Create the studies that contained these matrices and trees

* Description:

As of 20090209, 21/3793 matrices have not been loaded.  Why not?
Load them.
Four tree files have not been loaded. Why not?  Load them.

* Current status:
A - done
B - done
C - in progress
D - ten files referred to Bill.  He fixed seven.  See file "failed-matrix-1"
    What about the other 11 files?
E - 
F - Not started
G -
H - 
----------------------------------------------------------------
* Title: Can a user change their password?
* Owner: VOS
* See:  password-change-failed
* Label: PASSWORD
* Description: 

Fixed the Data Access Failure exception when changing the user profile. 
The problem was that the user ID was written incorrectly to a hidden 
input field in the form, and not updated in the User object. This has 
been fixed, and the view has been brought in line with the rest of the 
site a bit more.

* Current status: Resolved.

----------------------------------------------------------------
* Title: Hash passwords in database
* Owner:
* Description:

User passwords are stored in the database USER.PASSWORD field in plain
text.  They should be hashed.  

The initial user creation in database/initTreebase.sql will need to be
modified to match.

* See-also: hashed-passwords

----------------------------------------------------------------
* Title: Reviewer read-only access to studies
* See: reviewers
----------------------------------------------------------------
* Title: Taxon search
* Owner: MJD
* Label: TAXONSEARCH
* Description: To come
----------------------------------------------------------------
* Title: Taxon search using NCBI classification
* Owner:
* Description:  As per BP email of 20090208 "Re: Master task list right now"
<4BB519AD-8458-4F99-A2A8-0B2A3907D8C7@yale.edu>
----------------------------------------------------------------
* Title: Add help anchors in the search area
----------------------------------------------------------------
* Title: TaxonlabelGroup_TaxonLabel table misspelling
* Owner:
* Description:

SVN revision @6081 corrected a misspelled table name in the code:
TAXONLABELGROUP_TAXONLABEL was misspelled as
TAXONLABELROUP_TAXONLABEL. (Missing letter 'G')

Changing the code is not enough, because the existing schemas have the
misspelled name.  Rename the tables, and do whatever else is necessary
to fix this problem.

----------------------------------------------------------------
* Title: TB1 importation checking
* Owner: MJD
* Label: CHECKING
* Description: 

Use the "check" program at

    treebase-core/src/main/perl/check/check

to check the consistency of the imported data.  Resolve any
inconsistencies that are reported, or make an informed decision to
postpone them.

----------------------------------------------------------------
* Title: Search interface cleanup
* Label: SEARCHCLEANUP
* Owner:
* Subtasks:
  A: Make sure that table headings in summary and search results pages are consistent
  B: The display in search/study/trees.jsp has a spurious "1"
  C: Delete the rest of the class="bvnote" sections
  D: RHS menu is broken in search area
  E: Main tabbed menu is missing in search area

* Description
----------------------------------------------------------------
* Title: Searching fixes
* Label: SEARCHFIXES
* Owner: 
* Subtasks:
  A: Report misspellings in tree topology searches, or at least don't fail
  B: Specify page length in search results
  C: Matric text searches: character names, char. state names, char set names
  D: Clicking column heads to sort search results breaks sometimes
  E: Implement matrix "download" links; add to matrix.jsp
  F: Implement tree "download" links
  G: Character data in matrix row view should be adjustable
  H: Row segment display "not imeplemented" header is misformatted
  I: Test to make sure empty search boxes are gracefully handled
  J: Missing form name causes NPE in classes derived from SearchController

----------------------------------------------------------------
* Title: Resolve symbolstring / elementstring confusion
* Owner:
* Description: 

domain.MatrixRow has both getElementAsString and getSymbolString.
What is the difference between these?  Are they used consistently
throughout the code?
----------------------------------------------------------------
* Title: Resolve TB1_STUDYID / ACCESSIONNUMBER confusion
* Owner:
* Description:

The STUDY table has fields TB1_STUDYID and ACCESSIONNUMBER.  These
represent the same thing.  TB1_STUDYID should be eliminated everywhere
in favor of ACCESSIONNUMBER.  Make sure the data is copied over from
one column to the other!

----------------------------------------------------------------
* Title: Set test flag in uploaded submissions
* Owner: VOS
* Label: TESTFLAG
* Description:

During the test phase, studies submitted through the web interface
should get their SUBMISSION.TEST values set to 1.

VOS added getTest()/setTest(Integer pNewTest) to Submission, this was
necessary because the schema now has a non-nullable TEST column
as per TESTFLAG task. This fixed the Data Access Error. The upshot
is that Submissions are now "test" by default, so that they can
be filtered and deleted after beta testing.

* Current status: Resolved.

----------------------------------------------------------------
* Title: Document parser system
* Owner:
* Description:

MJD wrote a recursive-descent parser system (in
org/cipres/treebase/auxdata) to handle Bill's Dump.txt file.  The
parser system has no documentation.
----------------------------------------------------------------
* Title: Thank Rod Page
* Owner: Val
* Description:

Val should thank Rod Page for citation parsing etc.

----------------------------------------------------------------
* Title: Miscellaneous Bill Piel requests of 20080808
* Owner:
* Description: 

Bill sent mail on 20080808 with a number of requests.
Make sure these are properly addressed.

* See also: bill-20080808
----------------------------------------------------------------
* Title: Set up bug-tracking system
* Owner: Vos
* Label: BUG
* Subtasks:
  A: Get bug tracker
  B: Arrange for bugs to be submitted by email
  C: Have Bill set up forwarding address bugs@treebase.org
  D: Test

* Dependencies:
  A -> B
  B -> C
  B -> D

* Current status: Resolved.
* Description: 

Created the following issue trackers on sf.net: "bugs", "feature
requests", "patches" and "support requests". The plan is that users/
beta testers will submit their issues to these themselves, unless
they can't (e.g. don't understand sf.net interface, are lazy) in
which case they might email them to bugs@treebase.org, which VOS
will monitor and submit by hand.
----------------------------------------------------------------
* Title: Simple tree-delivering URL-API
* Owner: 
* Label: TREE-API
* Subtasks:
* Dependencies:
* Description:

"So that we can use a non-crippled version of PhyloWidget on our
search page" as per BP email of 20090208 "Re: Master task list right
now" <4BB519AD-8458-4F99-A2A8-0B2A3907D8C7@yale.edu>
----------------------------------------------------------------
* Title: Backport TB2 taxon intelligence database to TB1
* Label: TAXON-BACKPORT
* Owner:
* Description: 

  "I will need a taxon-intel dump from TB2 after the beta-testing is
   over but before the final TB1 data migration happens. This is so
   that I can bring the TB1 taxon-intel up-to-date with the TB2
   taxon-intel before I create new TB1 taxon-intel records and before
   I do a last migration to TB2." 

     -- BP (Re: Master task list right now) 20090208
        <4BB519AD-8458-4F99-A2A8-0B2A3907D8C7@yale.edu>


* Requirements: *BETA

----------------------------------------------------------------
* Title: Garbage-collect unused objects
* Owner:
* Last-modified: 20090305
* Description: 

*POP may have (and probably has) created unnecessary objects.  For
example, there may be taxonlabels that are referred to by no study in
the database, or studies (such as #22) that should not be there at
all.

At best, these objects take up space for no benefit.

They slow down bulk data operations, confuse debugging attempts, and
they may cause some queries to produce incorrect results.  For
example, as of 20090305 there is, for each TL object legitimately
attached to study 1121, a dupliucate TL that is not linked from the
study, but which links *to* that study.  Consequently, 

       select ... from taxonlabel where study_id = 1121

produces 92 items, but it should produce only 46.  

MJD has plans for a mark-and-sweep garbage collector program.

----------------------------------------------------------------
* Title: Correct broken taxonlabelsets
* Owner:
* Last-modified: 20090305
* Description: 

Preliminary consistency checking indicates that *POP did not create
the taxonlabelset objects properly.  For example, as of 20090305, the
following condition obtains:

  - S391 contains Tr3338
    - Tr3338 is in Tb5235
      - Tb5235 points to Tls9884
    - Tr3338 contains Tn240393
      - Tn240393 links to Tl105859
        - Tls9884 does *not* contain Tl105859

Figure out what the problem is here.  Fix it.
Find all other broken taxonlabelsets.  Fix them too.

----------------------------------------------------------------