Caitlin,
The reason for establishing a team and confirming the responsibilities is to ensure that all areas of the company have representation (EG: Maintenance, Cleaning, Procurement, New product development, despatch, laundry, lab) as well as production. If you are happy that you, the boss and the manager understand all areas of the company sufficiently to be able to carry out effective hazard analysis then there is no problem with the team being only you 3 persons.
BUT: - if you don't feel confident covering all of the areas and continue anyway you may find that you have missed a potential hazard which later will cause you a lot of re-work. This is why often for the hazard analysis more people are involved after the initial analysis some of these people may not be required except for jobs in their specific area of responsibility
I hope this helps, just out of interest what sort of company do you work for and how big (no. employees)
James
"arguing with an auditor is like wrestling with a pig in mud, eventually you realise that the pig enjoys it"