To better constrain and quantify the composition of material in the diffuse interstellar medium (ISM), absorption spectra between 3600 and 2700 cm-1 (2.8 and 3.7 microns) have been taken of objects which have widely varying amounts of visual extinction along different lines of sight. The spectra of these objects contain a broad feature centered at approximately 3300 cm-1 (approximately 3.0 microns), attributed to O-H stretching vibrations, and/or a feature near 2950 cm-1 (3.4 microns) attributed to C-H stretching vibrations. The lack of correlation between the strengths of these two bands indicates that they do not arise from the same molecular carrier. The features in the 3100-2700 cm-1 (3.2-3.7 microns) region fall into one of two classes. We attribute the first class of features to material in the diffuse ISM on the basis of the similarity between the band profiles along the very different lines of sight to Galactic center source IRS 7 and VI Cygni #12. Similar features are also reported for Galactic center source IRS 3, Ve 2-45, and AFGL 2179. Higher resolution spectra of the objects OH 01-477 and T629-5, which are known to be M stars, are dominated by a series of narrow bands in this region. These bands are largely due to OH in the stars' photospheres. While the spectra of OH 01-477 and T629-5 are likely to contain C-H absorption from diffuse ISM dust, the strength of the overlapping photospheric OH features presently prevents us from quantifying the depths of the interstellar C-H feature towards these objects. The interstellar feature for Galactic center source IRS 7 has subpeaks near 2955, 2925, and 2870 cm-1 (+/- 5 cm-1), which we attribute to C-H stretching vibrations in the -CH2- and -CH3 groups of aliphatic hydrocarbons. These band positions fall within 5 cm-1 of the values normal for saturated aliphatics. The absence of a distinct band near 2855 cm-1 suggests that the material contains small amounts of electronegative groups like -O-H or -C triple bond N. The relative strengths and profiles of the 2955 and 2925 cm-1 features towards five objects suggests an average diffuse ISM line-of-sight -CH2-/-CH3 ratio of about 2.5, indicating the presence of relatively complex organic materials. The strengths of the subpeaks at 2925 and 2955 cm-1, due to -CH2- and -CH3 groups, respectively, correlate with visual extinction, strongly suggesting that the C-H stretching band is a general feature of the material along different lines of sight in the diffuse ISM. We find average ratios of A nu/tau(2925 cm-1) = 240 +/- 40 and A nu/tau(2955 cm-1) = 310 +/- 90 for the objects we have observed. We deduce that 2.6%-35% of the cosmic carbon in the ISM is tied up in the carrier of this band with the most likely value falling near 10%. The interstellar C-H band is remarkably similar to the feature in lab residues produced by irradiating analogs of dense molecular cloud ices. This is consistent with a model in which the hydrocarbon component in the diffuse interstellar medium consists of complex hydrocarbons containing aliphatic side chains and bridges which are produced in dense molecular clouds and subsequently modified in the diffuse medium.