His wife of 42 years, the former Mary Katherine “Kay” Watson, died in 1998. Survivors include his wife of 22 years, the former Darian Dreyfuss Olsen of McLean; five children from his first marriage, Katherine Downs and Carol Downs, both of Roslindale, Mass., Christine Mann of Richmond, Calif., James Anthony Downs Jr. of Sherborn, Mass., and Paul Downs of Penn Valley, Pa.; three stepchildren, Howard Olsen of Burlingame, Calif., Spencer Olsen of Waxhaw, N.C., and Jennifer Yeamans of Livermore, Calif.; 13 grandchildren; and three great-grandchildren.