While the number of women receiving bachelor's degrees in physics in the U.S. continues to increase, the number of men is increasing faster. The representation of African Americans, Native Americans, and Hispanics in physics continues to be extremely low. The American Institute of Physics (AIP) collects data on the representation of women and members of other under-represented groups in physics at all levels, from high school students to faculty members. However, the data on representation do not tell the whole story. Data from a global survey of physicists show that women do not have access to the same opportunities and resources as men do. Data from astronomers show that less than ideal relationships with advisors and the two-body problem contribute to working outside the field, and that these factors affect women more than men. AIP has just started data collection efforts on the factors that contribute to the low numbers of African Americans in physics at the undergraduate level. These types of data are essential for designing programs that will increase the representation and retention of women and minorities in physics and astronomy.