m = Factory(:milestone) just wasn't working. Don't know why.
I had the following factories:
factories.rb
Factory.define :milestone do |m|
 m.association :rulemaking
 m.association :phase
 m.association :activity
 m.association :date_category
 m.milestone Date.today
 m.association :revision#_number, :factory => :revision
end
 Factory.define :revision do |r|
   r.revision_number { Factory.next(:integer)}
   r.revision_date Date.today
 end