Admit it -- Accept it
Say it with me: size does matter. As much as we would like to believe it doesn't, it does. Now, the sheer size of your list doesn't stand on its own as a measurement of success for your email program, but list growth can tell you that you have content or information that people want, continue to get and perhaps spurs them to tell others, resulting in increased subscription numbers. (Clearly this argument does not hold true if you just bought a million email addresses from some guy in India -- we're talking legitimate, real list growth.) The fact of the matter is that if you are losing subscribers faster than you are gaining them, you have some issues with your email program that need to be addressed. So don't discount your list size as a metric for success -- admit that it is important, accept that others think so, too (though for different reasons I am sure) and use that information to your benefit.
Control it -- Own it
Setting proper expectations for list growth is your responsibility. Often those who control our budgets don't really care about the details of how you achieve what you achieve -- what's important is that you meet your goals and objectives. And if you are an email marketer, you either have a goal to minimize attrition or a growth goal for your subscriber base. Be sure that you are setting the goals (or at least influencing them) around growth. They should be realistic numbers that align with the addition of quality subscribers who want to engage with your content and your brand and will have a positive impact on your other performance metrics. If you are just adding people for the sake of hitting a number, the rest of your metrics will suffer, I promise.
Live it -- Love it
Live the email experience as your customers do. How do they get subscribed, do you have a social presence you can leverage, what points of contact do they have offline that allow for natural email acquisition -- and, most important, do your current subscribers love your content enough to evangelize on your behalf? However you decide to go about acquisition and backfilling, be sure you love it and are behind it 100% -- because if you are not, you will not put your focus on its success. Because in the end, though you may not like it: size does matter.